Changelog from AN-2021-07-29:
- star: A single Linux xattr entry with a zero length value did disable
all Linux xattrs for the related file. This is now handled correctly.
Note that setfattr(1) on some Linux versions does not support to set
empty xattr entries, but star on the same system can extract files
with empty xattr entries.
Thanks to a report from Dennis Katsonis
- star: star -version now prints a new version date
Changelog from AN-2020-11-04:
- Makefile system: include/schily/nlsdefs.h no longer by default defines
the macro __() because this is in conflict with definitions that are
present in the system include files from newer HP-UX versions.
Thanks to Rudi Blom for reporting.
- star: The ACL support code and the Linux xattr code had a typo related
to very long path names. As a result, ACLs and Linux xattrs did not work
if the path name to a file was longer than PATH_MAX.
Changelog from AN-2020-11-25:
- Makefile System: Added support for MacOS on arm64
Thanks to a hint from Ryan Schmidt from macports
Note that due to outstanding replies to recent changes in configure,
it could up to now not be verified that all configure tests now work in
a way that results in correct overall results. See below for an in
depth report on the changes.
- Makefile System: autoconf (config.guess & config.sub) now supports
the new arm64 Apple systems.
Thanks to Ryan Schmidt from macports for provinding the needed uname(1)
output.
- Makefile System: Added a new shell script "autoconf/uname" that helps
to create shell scrips that allow to emulate an alien host system in
order to test the correct behavior of configure.guess and configure.sub
on the main development platform.
This helps to adapt configure.guess and configure.sub to new platforms
in the future.
- Makefile System: The new clang compiler as published with the upcomming
ARM macs has been preconfigured with
-Werror -Wimplicit-function-declaration
as the default behavior and thus is in conflict with the existing base
assumption of the autoconf system that minimalistic C-code used for
compile/link allows to check for the existence of a specific function
in libc without a need to know which system #include file is used to
define a prototype for that function.
This clang version, as a result of this default, behaves like a C++
compiler and aborts if a function is used with no previous function
prototype. This caused most of the existing autoconf test to fail with
error messages about missing prototypes.
We implemented a workaround using these methods for the identified
problems:
- Most of the exit() calls in the various main() functions have
been replaced by return() to avoid a need to
#include <stdlib.h> in special since these test may be the
case for layered tests that #include files from the higher
level parts.
- Many autoconf tests programs now #include more system include
files, e.g. stdlib.h and unistd.h to avoid missing prototype
errors. This cannot reliably be done in tests that are used as
a base for higher level tests where the high level test
#includes own system include files, since older platforms do
not support to #include the same file twice.
So this is tricky...
- A test for a Linux glibc bug caused by incorect #pragma weak
usage inside glibc that prevents one or more functions from
ecvt()/fcvt()/gcvt() from being usable outside glibc now uses
hand-written prototypes for some of the libc interface
functions in order to avoid using the system includes. If we
did not do that, we could not use ecvt()/fcvt()/gcvt() on
MacOS anymore.
Thanks to Ryan Schmidt from macports for reporting and for the given
help that was needed for remote debugging.
Please send the needed feedback on whether the current state of the
configure script results on correct autoconf results on the M1 Macs.
New features with AN-2020-09-04:
- autoconf: congig.guess: FreeBSD on 64 bit ARM returns arm64 from
uname -m; this was previously not supported and rejected by config.sub.
We now convert "arm64" into "aarch64" in config.guess to get the usual
expected results.
- Makefile system: RULES/rules.env The environment variables FIGNORE,
LD_LIBRARY_PATH LD_LIBRARY_PATH_32 and LD_LIBRARY_PATH_64 are now
unexported from the enviroment.
In special FIGNORE is dangerous, as it is frequently used by bash
users but tells ksh93 to modify it's behavior with "echo *" and
this may cause strange things with our makefiles in case that
/bin/sh is ksh92. This applies e.g. to Oracle Solaris 11 and
OpenSolaris.
New features with AN-2020-10-09:
- autoconf: Added support for newer HP-UX versions on Itanium.
Thanks to Rudi Blom for reporting and making a change proposal.
New features with AN-2020-07-18:
- star: star could dump core if it was used as "star -t ..." or.
"star -x ..." while being in a UTF-8 based locale and trying to deal
with extremely long pathnames (more than PATH_MAX) in the archive.
This bug was caused by the dummy conversion routines _to_utf8() or
_from_utf8() that did not stop after "tolen" bytes (the current size
of the dynamically growing path structure) have been copied.
This bug has been introduced in 2018 when the dynamic path name
variables have been introduced together with support for extremely
long path names.
- star: lpath_unix.c and lhash.c renamed a local variable buflen to bflen
to avoid a gcc shadowing warning with the rest of star.
- star: star.c fixed some fallthrough warnings from lint.
- star: The FIFO code (which is 30 years old) did use an int for the size
which historically was OK, but this did limit the size of the FIFO to
2 GB. Now with modern tape drives that are really fast, a FIFO with
2 GB would only give a tape streaming reserve for approx. 8 seconds,
which is not sufficient. Approx. 30 seconds reserve are recommended.
With modern tape drives, this would need approx. 8 GB of FIFO. Be.
however careful not to use more than half of the real RAM available
in the whole system for the star FIFO.
Note that this change induced the need for a lot of derived changes
in approx. 1000 lines of code spread over the whole star project and
for this reason, it is advised to carefully test the new version and
to report if problems occur.
New features with AN-2020-08-12:
- autoconf: added a new test for the existence of <sys/auxv.h>
- autoconf: added new tests for Linux getauxval() and FreeBSD elf_aux_info()
- libschily: getexecpath.c now uses getauxval() on Linux and elf_aux_info()
on FreeBSD. This was needed since readlink("/proc/curproc/file") on
FreeBSD returns random values for hardlinked files, making it impossible
to use the result in order to find out which behavior variant of a
fat binary is requested.
- star: GNU cpio archives may contain longer path names than permitted by
the AT&T documentation. This did cause problems with our automated
recognition of swapped CPIO headers. We now use a more robust method.
- star: GNU cpio archives may contain longer path names than permitted by
the AT&T documentation. This could cause star to dump core because the
cpio module did not yet call the routine to grow our new dynamic path
store.
- star: older GNU compilers seem to decide that a division:
long long / long results in a long.
We now cast the divisor to long long to definitely get the right
result for a printf argument.
- star: changed a name in a parameter list for lreadlink() to avoid
a GCC shadow warning.
- star: The variable use_fifo is also needed if the FIFO is not
available (e.g. on DOS). The previous code did not compile on platforms
that did not support to implement the FIFO. We moved
extern BOOL use_fifo out of a #ifdef FIFO
Thanks to a hint from Albert Wik.
- star: The file README.otherbugs has been changed to match the
state of today and a geocrawler based URL now includes an
archive.org prefix as geocrawler no longer exists.
Changelog from AN-2020-07-01:
- star: Backslashes are now honored when looking for the substitution
separators in options like -s /from/to/ or -pax-s /from/to/
Previously, -s '/xxx\/yy//' was interpreted in a way that made
the **to string** "yy" and the **from string** "xxx\" instead of
making the from string "xxx\/yy" and the to string empty.
- Remove pkgsrc patches (merged upstream)
- Move OS specific helper code to devel/smake/Makefile.common
Changes from AN-2020-05-25:
- libmdgest: Added a "defined(_M_ARM)" check for MSVC byte order
definitions (byte_order.h).
Thanks to "Excalibur" for reporting.
- star: SmartOS supports aclutils.h as published include file (what we
intented to include in Schillix-ON as well, since it is needed for
libsec) and aclutils.h includes ctype.h. We thus need to
#include <ctype.h> before star.h in acl_unix.c
The problem occurred because we did already have a aclutils.h related
autoconf test, even though Schillix-ON does not yet ship aclutils.h
Thanks to Michael Bäuerle from pkgsrc for reporting.
- star: The subst command (since 2018) malloc()s the space for a copy
of the old string in case that the string Is too long for the static
storage. It did however still use the static buffer instead of a
pointer to the allocated space. We now use the allocated space if
needed.
- star: The subst command for pax now supports the \1, \2, ... escapes
for \(...\) selections in the from pattern, like it is used by sed(1).
Thanks to Kristyna Streitova for reminding on this.
- star: A new unit test for this new feature has been added, using the
example from the SUSE bug-report for the above substitute command.
- star: A new subst option -pax-s has been added that supports sed(1)
like substitutions. The old option -s continues to support change(1)
like substititions.
NOTE: Both pax -s and star -pax-s silently fall back to change(1)-like
substitutions in case that a target platform does not support "regex"
in libc.
- Check which systems need and support processing tables with tbl
The default case matches for NetBSD (no processing)
- Man pages are installed into native OS sections
PLIST must match this logic (use variables)
Added SPECIAL_PERMS for config file (no more "wrong permissions"
warning if installed as nonprivileged user.
Changes from AN-2020-03-11:
- configure: The autoconfiguration now has an enhanced test for waitid()
that was needed since Mac OS is still not POSIX compliant and returns 0
instead of the signal number for a process that has been killed by a
signal. MacOS did pass the POSIX certification as a result of a missing
test for that problem.
Since every vertified OS needs to run an annual refresh the
certification with recent versions of the test, I guess that
Mac OS (Catalina updates) may become more compliant witin a year.
- psmake: The portable bootstrap compile environment for smake missed
a symlink for unsetenv.c to libschily since the related code has been
moved to libschily in 2018. This prevented compilation on IRIX.
Thanks to Kazuo Kuroi for reporting
- librmt: the code has been modified to allow librmt to be finally
compiled without a need to link against libschily. This makes it
easier to link ufsdump/ufsrestore from OpenSolaris against librmt.
A call to errmsg() was replaced by a call to errmsgno() that has
a private implementation in librmt already.
- librmt: the man pages have been modified to make sure that man2html
is able to correctly create links to referenced other man pages.
- libstreamar: streamarchive.4 now mentions that the POSIX.1-2001
tar extensions are based on a Sun Microsystems proposal from 1997.
- star: the man page now mentions that it is preferred to use the
source from the schilytools instead of a star specific release.
Changes from AN-2020-03-27:
Changes from AN-2020-04-18:
- librmt: A few small enhancements have been added to the man pages.
This was a result of a code review for SchilliX-ON by
Eric Ackermann
- star: The unit tests port/lpath.sh did not work on IRIX because
the "rm" command on IRIX is buggy and does not remove long path
names (path names longer than PATH_MAX). We now use a recursive
shell function to remove the long path.
- star: The new shell function in port/lpath.sh did crash Linux in
case that /bin/sh is "dash". "dash" is not POSIX compliant and
fails to support arbitrary long working directory names. Whe therefore
try to run the long path removal with "bosh", or at least "bash".
Only if none of both is present, we fall back to the system default
shell "/bin/sh".
Changes from AN-2020-05-11:
- Makefile system: A new version of the BSD make (bmake) program fixed
a bug in pattern macro substitution, so we are now able to detect
BSD make and to read BSD make program specific rules.
This could in theory allow us to support BSD make in the future,
but...
Note that we on the other side discovered a new bug with pattern
macro substitution in bsd make: The substitution:
$(FOO:%=bar/%)
is replaced by "bar/" with an empty "FOO", but of course, with an
empty FOO, the substitution should be empty as well.
This second bug (above) was fixed on May 6th, but we do not yet have
all needed make rules and we do not know whether other bugs may still
prevent the usability of BSD make. Supporting BSD make will be hard
as BSD make does not support pattern matching default rules and this
is important for placing the .o files into a sub-directory.
Also note that the portable program source for "bmake" from "pkgsrc"
is 2 years old and thus currently cannot be supported at all. If
you like to experiment on your own, you need to get this version:
http://crufty.net/help/sjg/bmake.html
see
http://www.crufty.net/ftp/pub/sjg/bmake.tar.gz
and replace the newer files from the netbsd.org CVS tree by hand in
order to fix the first and second mentioned pattern macro substitution
bug.
- Makefile system: RULES/MKLINKS was enhanced to create a new symlink
RULES/r-bsdmake.tag that points to RULES/r-make.tag
- Makefile system: The archive makefiles.tar.bz2 has been added to the
schilytools tree to allow easy reuse of the makefile system for own
projects.
New features with AN-2020-02-11:
- star: fixed typos in star.1 and star.4
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
- star: fixed typos in header.c
Thanks to Eric Ackermann for reporting
- star: translated some German comment to English in diff.c, extract.c,
list.c, remove.c, restore.c, fifo.c
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
- star: create.c: #ifdef __what_people_would_expect__ changed to:
#ifdef __wrong_but_what_people_would_expect__ for better readability.
- libschily: new man pages starthandlecond.3 and unhandlecond.3
- libschily: handlecond.3 and raisecond.3 now correctly mention
handlecond() & raisecond() instead of handle()/raise(). The old names
from 1980 had to be renamed because os an unfriendly actt from the
C standard commitee.
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
- libschily: various small fixes in various man pages from libschily.
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
pkglint --only "https instead of http" -r -F
With manual adjustments afterwards since pkglint 19.4.4 fixed a few
indentations in unrelated lines.
This mainly affects projects hosted at SourceForce, as well as
freedesktop.org, CTAN and GNU.
Based on Schily Tools Release 2019-12-05.
AN-2019-11-11:
- libschily: the new functions wcastoi wcastol wcastolb that have been
added with the previous release have been forgotten in the linker map
file.
- star: When using star -copy, the default is now -no-secure-links.
This is what users like since they are copying their own data that
per definition cannot be untrusted external data.
- star: star -diff is now "more smooth" to use, when the archive type
is GNU TAR and the "atime" or "ctime" properties are not present
for a specific file. Before, Jan 1 1970 was the assumed time in
the archive.
- star: star -diff now prints a diff statistics summary.
- star: New version date
AN-2019-12-05:
- include/schily/stdio.h: We no longer define getc_unlocked() on Solaris
in case that it is defined already. This was needed since Oracle
Solaris 11.4 compiles 64 bit by default and made stdio less opaque
than before - this result in the same bahavior as for 32 bit programs.
This means, we no longer need to use tricks to get fast access to
stdio.
Thanks to Rolf Dietze for reporting.
- libschily: the function getargs() and similar (from the option parser
group) fixed a bug that prevented the usability of the official method
to include the char '+' inside option names by quoting the character
in the option name used in the format string.
This bug has been fixed in order to be able to support "star -dump+ ..."
- libshedit: because of the changes in Oracle Solaris 11.4, we need to
#undef FAST_GETC_PUTC in order to avoid to use getc_unlocked().
Thanks to Rolf Dietze for reporting.
- libfind: A new function walksname() has been added to set up the
internals in "struct WALK" for external path names. This feature is
needed to support "star -c list=names -find ...".
- libfind: The functions fetchdir()/dfetchdir(), sortdir(), cmpdir() now
use size_t instead of int for the entry number count and the allocated
size.
This breaks binary compatibility in 64 bit mode for those users only,
that use the interfaces fetchdir()/dfetchdir(), sortdir(), cmpdir().
The only application that currently uses these interfaces is "star".
Simple users of libfind and programs compiled as 32 bit programs
are not affected by the change.
Warning: packetizers need to take care of the changed name of the
shared library file for libfind.
- libfind: Because of the binary incompatible change, we incremented
the major library version number from 3 to 4.
WARNING: This affects only distros that ship 64 bit versions of star
using a shared version of libfind.
Since Solaris defaults to 32 bit programs, this is expected to only
affect Linux distros.
- libstreamar: unicode.c has been changed to make sure that is works even
in case that the "len" parameter is an unsigned.
- libstreamar: to_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- libstreamar: from_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: The new-volume-script= script is now called at the end of
_every tape_, even at the last one. This now allows to have a script
that automatically renames a standard archive name into numbered
archive names when a tree should be splitted into chunks, e.g. with
this command line:
star -c f=/tmp/v.tar tsize=1G new-volume-script='sh -c \
"mv /tmp/v.tar /tmp/v$(($1-1)).tar" nv' .
you get partial archives named /tmp/v1.tar ... /tmp/vn.tar
Note that this would not work in case that "sh" is the historical
Bourne Shell, e.g. as seen on Solaris 10. This is because
"shell arithmetics" is a feature that was not present in the historical
Bourne Shell.
- star: The man page now mentions that "star cli=tar ..." works as an
alias to "star cli=suntar ...".
- star: The man page now mentions that "star cli=gnutar ..." does not
impement the bugs from the GNU option parser.
- star: Is "star cli=xxx ..." is used, then star now calls
set_progname(xxx) and thus prints error messages labelled with the
command line interface name.
- star: In "pax" command line mode, star now uses -no-fsync as default
in order to behave the same way as the closed source Solaris pax
does.
- star: Star now allows to combine the option list=names with -find
This is useful, whenever a list of filenames is longer than ARG_MAX and
thus cannot be used in the star command line after -find.
Here is an example on how to use the new feature in a mercurial.
based development environment:
(hg status -a; hg status -m; hg status -c) | sed -e 's/^. //' > .hgfiles
star -c list=.hgfiles -find -mtime -100 > /tmp/new.tar
The first command puts all files under control of mercurial into
thw file ".hgfiles" , while omitting the compile results from the
file ".hgfiles".
The second command takes that list and puts only those files into
the archive that have been modified during the past 100 days.
- star: A new option "-dump+" allows to have the "SCHILY.volhdr.hostname"
meta data field filled out even when not making dumps with an
associated dump level. This helps to remember the origin of tar
archives.
- star: unicode.c has been changed to use the same comment as unicode.c
in libstreamar.
- star: unicode.c has been changed to make sure that is works even.
in case that the "len" parameter is an unsigned.
- star: to_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: from_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: due to the incompatible interface change in libfind (see
above), several changes have been introduced in order to use size_t
instead of int for fetchdir()/dfetchdir(), sortdir(), cmpdir().
This finally permits star to be able to make use of the current POSIX
maximum meta data size for so called "pax" archives (TAR archives
with POSIX.1-2001 meta data enhancements derived from the Solaris 7
concept for tar from 1997). Previous versions of star have been
limited to data no longer than 2 GB (even though the star main code
supports 8 GB file size), this version now supports 8 GB for the.
sum of all meta data for a single file in case a 64 bit binary from
star is used.
- star: the fifo now has a new variant of te function runfifo() that
is used by the new fifo(1) standalone program. See below.
- star: The README now mentions the new "star cli= ..." method and
the fact that star deals with infinite path name length.
Based on schilytools release from 2019-10-25.
Changelog
=========
- libstreamer: Added new keywords "fsdevmajor" and "fsdevminor" as
there are too many incompatible st_dev binary formats.
These keywords are not yet created, but they are understood.
This permits to switch to the new format (fade out "dev") later.
- star: Added new keywords "SCHILY.fsdevmajor" and "SCHILY.fsdevminor"
as there are too many incompatible st_dev binary formats.
These keywords are not yet created, but they are understood.
This permits to switch to the new format (fade out "SCHILY.dev") later.
- star: The amount of shared mamory used by default has been reduced to
1 MB on Ultrix. These are old small machines, that do not support 8 MB.
Based on Release 2019-10-07.
Changelog
=========
- configure: Some shells report a syntax error with "< file (cmd)"
and need the redirection statement to be *after* the command. Our
changes to support the V7 shell by adding round braces caused ash
variants like "dash" to fail.
Thanks to Harald van Dijk for reporting
- cont/cc-config.sh: canged some :>some-file statements into
(:)>some-file. they have meen missed when introducing work arounds
for the V7 Shell on Ultrix that does not support I/O redirection
for builtin commands.
Thanks to Robert Clausecker for reporting
- libschily/resolvepath.c: resolving a symlink that points to another
symlink that points to itself, caused a coredump as a result from an
endless recursion.
We now detect this situation and abort the check before the endless
recursion causes a stack overflow. A symlink that directly loops
is immediately stopped. A longer symlink loop chain over more than one
symlink can only be detected by the recursion nesting level and is
aborted after a nesting level of 1024 has been reached. This works
under the assumption that the minimum stack size is more than
1024 * PATH_MAX and that there is no useful directory path with more
than 1024 symlinks in the path.
----> This problem affected star and SCCS.
Thanks to Philipp Wellner for reporting
- star: Added a hint to the man page that helps to find pkglist= as a.
similar option to list=
- star: The new method to avoid extracting symlinks that point outside
the star working directory that has been introduced in October 2018
could cause a core dump if a symlink is checked that points to
another aready existing symlink that points to itself. This was caused
by a problem in libschily/resolvepath.c, see above.
Thanks to Philipp Wellner for reporting
- star: The option -no-secure-links now may be configured as a global
default via the tag STAR_SECURE_LINKS= in the file /etc/default/star
and as a private default via an environment of the same name.
If the value for this tag is 'n' or 'N', -no-secure-links is made the
default, any other value sets the option -secure-links as the default.
This may be useful for sysadmins that frequently use star to copy
installation specific files, but it is risky in case that alien TAR
archives are imported. The good news is that this permits users to
switch to the old star behavior where no checks for risky links
existed.
Thanks to Dennis Clarke for reporting
- star: A new enviroment STAR_NOHINT has been introduced to supress
hint messages that are otherwise seen in case STAR_SECURE_LINKS or
STAR_FSYNC is in the environment or in /etc/default/star
- star: New version date
Switch to latest distfile.
Update PLIST.
Replaced MESSAGE with share/doc/star/INSTALL.pkgsrc.
Changelog
=========
Release 2019-03-29:
- libschily: the lutimens() emulation no longer returns ENOSYS in case
that the lstat() call fails, since this error code may e.g. be
ENAMETOOLONG and used as an important indicator for long path name
handling.
This bug caused incorrect behavior when star extracted long pathnames
on a platform without utimensat().
- star: The code restructuring to openat() from Summer 2018 did not
only cause a noticeable speed up even when not using -find, it at
the same time resulted in a wrong error message when a file type
was seen that is not archivable (e.g. a socket while using the USTAR
format). The correct message was something like "Unsupported filetype",
while the bug caused a "file not found " message. This has been fixed
by adding a new parameter "fd" to the function stat_to_info().
- star: header.c::get_xhtype() we did forgot to initialize:
finfo.f_devminorbits
and
finfo.f_xflags
This is now done.
- star: The USTAR format now is able to create base-256 values in
the field "t_devmajor". "t_devminor" did already support base-256.
- star: The GNU tar format now is able to create base-256 values in
the fields "t_devmajor" and "t_devminor".
- star: The man page star.4 now mentions which fileds may have
base-256 numbers.
- star: The man page star.4 now longer contains the doubled "field field"
in the explanation of the added UID/GID number for ACLs.
- star: The man page star.4 now has a better description for the additional
numeric fields in the ACL entries to prevent GNU tar from continuing
with it's incompatible ACL implementation.
- star: New ACL reference archives for the ultra compact format, that has
been defined together with the libachive people, have been added to the
directory testscripts/:
acl-nfsv4-compact-test.tar.gz
acl-nfsv4-compact-test2.tar.gz
acl-nfsv4-compact-test3.tar.gz
acl-nfsv4-compact-test4.tar.gz
acl-nfsv4-compact-test5.tar.gz
- star: Linux now uses the new <linux/fs.h> instead of <ext2/ext2_fs.h>
to retrieve file flags.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for the following new BSD Flags:
compressed hidden offline rdonly reparse sparse system
has been added.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for the following new Linux flags:
dirsync nocow notail projinherit topdir
has been added.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for reading non-comliant libarchive tarballs that use.
"securedeletion" or "journal"
instead of the documented text has been added. This was used by
libarchive until March 20, when libarchive has been fixed. We introduce
the above names for compatibility with old TARs created by libarchive.
- star: When an "old star" archive is read and this archive is from a system
like FreeBSD with non-continous minor bits, star no longer warns unless
there is a device file in the archive.
- star: unit tests: the scripts now contain
LC_ALL=C export LC_ALL
instead of just LC_ALL=C to make the environment exported.
- star/libstrar: The case where iconv() returns -1 and sets errno to E2BIG
is now handled correctly.
- star: The testscripts/ directory now contains the tar test archives
from the portability tests from Michal Gorny. Check
http://cdrtools.sourceforge.net/private/portability-of-tar-features.html
for an updated (to match star-1.6) variant of the results from the portability
tests from Michal Gorny.
- star: older versions of star did not print the messages:
"WARNING: Archive is 'xxx' compressed, trying to use the -xxx option.\n"
in case that the option -print-artype was in use. Later versions that
added support for more than "gzip" and "bzip2" forgot to add the same
exception for the new compression methods. We now ommit this message
for all compression types.
- star: Support for the "lzma" compression has been added.
- star: Support for the "freeze2" compression has been added.
- star: The compression method name list did not include the text "zstd".
We added the missing text.
- star: The hint messages that are printed when a compressed input archive
is not seekable did forgot to mention "lzip" and "zstd". We added the
forgotten messages.
- star: New unit tests check whether star is able to auto-detect and auto
decompress various compression formats.
- star: The unit tests now include the portability tests from Michal Gorny
- A new option cli=name (must be argv[1]) allows to select all
supported command line interfaces (star, suntar, gnutar,
pax, cpio) when called as star.
This is needed to be able to test all command line interfaces from
out unit tests since star is not installed in this case and a selection
from argv[0] would not work.
- star: The version date is now "2019/03/20"
Release 2019-04-29:
- libfind/star: the verbose listing code has ben restructured to have
the file permissions in the same string as the file type.
This is needed to implement the POSIX pax listformat interpreter
format %M in future.
- libfind/star: the verbose listing no longer prints "l" for mandatory
record locking for non-directory type file, but rather only
for plain files.
- star: The "gnutar" emulation now prints the --help output to stdout as
GNU tar does.
- star: The gnutar.1 man page now mentions that the GNU tar
options -g / -G did never work and it thus makes no sense to
implement them.
- star: very outdated code in list.c has been removed.
- star: Added new unit tests for incremental backups and restores.
This in special include tests that always fail with GNU tar
as GNU tar is not usable and never was usable for incremental
restores in case that the differences are more than trivial.
- star: new version date 2019-04-01
Release 2019-06-13:
- star: fixed a bug in the FIFO code that mainly happened on Linux (with
a 1000x higher probability than it happens on Solaris). The bug was.
caused, as a check for a flag has been done twice instead of only once
where it could change it's value between both locations. As a.
result, star reported "star: Implementation botch: with FIFO_MEOF"
as the tar side of the FIFO did sometimes not wait for the FIFO_IWAIT
state when called as "star -multivolume -tv f=... f=... ...
For this reason, star incorrectly got a wakeup at the wrong location.
The bug appeared in case that star -x/-t -multivol f=.. f=.. ....
has been called with very small tar archives.
Thanks to Heiko Eissfeldt for reporting.
- star: The FIFO code renamed the "flag" member of the "m_head" structure
to "gflag" for better readability.
- star: The debugging code in the FIFO has been enhanced to print the
names of the flag bits in addition to the hex values.
- star: A deadlock situation that happens once every 500000 tries on
Linux with multi-volume archives has been fixed.
The problem was caused by a complex condition where the get side
of the FIFO needs to check the EOF FIFO flag and the amount of data
available in the FIFO and then decide whether to wait for a wakeup or
not.
Since the EOF flag needs to be checked first, a context switch in
the get side of the FIFO could allow the put side to set the EOF flag
before the get side did check the fill ratio of the FIFO. This
caused both the get side and the put side to wait for a wakeup.
The new code introduced a new variable mp->mayoblock that is set by
the get side before checking for EOF. This new flags allows the
put side to know that the get side is just in a critical situation
and lets the put side wait until mp->mayoblock is no longer set,
which signals a stable state in the get side. This permits to
avoid the deadlock.
- star: Note that the FIFO has been initially written as a lock free
design in the late 1980's. This is to allow high portability to even
older UNIX versions. The star FIFO works on all UNIX variants that
support pipes and shared memory, which is e.g. the case for
SunOS-4.0 from 1988. At the time the FIFO has been designed, the
target OS did not support multi-CPU systems and problems in the
FIFO first appeared with massively faster multi-CPU systems around
y2000. The recently detected problems all have been triggered by a
different context switch behavior on Linux, even though they could
have appeared on any OS in case that many million tries are
attempted.
- star: bumped version to 1.6.1
Release 2019-07-15:
- star: the compress unit test no longer fails on Cygwin because the
gzip binary is compiled incorrectly and does not support LZW.
The related test is now skipped.
Thanks to Heiko Eissfeldt for reporting
- star: some new unit tests failed if the schilytools source tree has
been installed in a directory with spaces in it's name.
Thanks to Heiko Eissfeldt for reporting
- star: The unit tests for incremental backups include archives that
include userid/groupid and username/groupname that may not be
restorable on a different computer. We now ignore these ID meta.
data when comparing the results.
Thanks to Heiko Eissfeldt for reporting
- star: When comparing nanoseconds in time stamps, star now has a mode
that treats time stamps as equal in case that tv_nsec % 100 == 0
and the rest of the nano seconds is equal. This is needd on Cygwin
since NTFS counts in 1/10 microseconds since Januar 1 1601.
This applies to both star -diff and the "newer" check while extracting
files.
Thanks to Heiko Eissfeldt for reporting
- star: a new option diffopts=dnlink has been implemented to support
filesystems that do not follow the historic UNIX model for hard links
on directories.
Since Cygwin usually has a linkcount of 1 on directories, you need
to use "star -diff diffopts=!dnlink ..." if you like to diff the
meta data from a historical UNIX filesystem.
Thanks to Heiko Eissfeldt for reporting
- star: The incremental backup/restore tests now use.
"star -diff diffopts=!dnlink ..." to make them work on Cygwin.
Thanks to Heiko Eissfeldt for reporting
- star: star -c -H exustar -acl -xattr-linux .
did cause file not found messages from the attempt to archive the
Linux xattrs. This was caused by a change from July 2018 when trying to
optimize directory access in (non-find) create mode. The call to read
the Linux xattrs has now been moved to the location where in former
times the ACL code has already been moved.
The move is needed because there is no ACL/XATTR related function
that is similar to openat().
- star: A similar problem with get_xattr() did exist with star -diff
- star: star -diff did not compare ACLs since getinfo() no longer
includes a call to get_acl(). We now call get_xattr() and get_acl()
in diff.c
- star: The Solaris ACL interface now implelements lacl() / lacl_get() /
lacl_set() to support very long pathnames with ACLs.
- star: The Linux ACL interface now implelements lacl_get_file() /
lacl_set_file() to support very long pathnames with ACLs on platforms
that implement the withdrawn POSIX ACL draft.
- star: The Linux xattr interface now implelements.
llgetxattr() / llsetxattr() / lllistxattr() to support very long
pathnames with Linux xattrs.
- star: New version date
Release 2019-07-22:
- star: "pax -pe" no longer sets the variable "doxattr" as this caused
an error message:
"NFSv4 extended attribute files are not yet supported.\n"
that caused the OpenSolaris-ON "nightly" compilation to abort as a
result of that error and exit code != 0.
Release 2019-08-13:
- libschily: Various functions and *at() emulation functions call stat()
even though the caller does not know about that call.
This could cause a missbehavior in case that a file returns EOVERFLOW
with a normal stat(). We now compile these functions in unconditional
large file mode to overcome that problem.
The affected files are:
diropen.c lutimens.c findinpath.c linkat.c mkdirs.c searchinpath.c
resolvepath.c lchmod.c renameat.c.
Note that this problem affected star(1) on platforms that do not
fully implement all *at() interfaces, since star started to support
really long path names in July 2018.
- star: Star did not compile on platforms without ACLs anymore since
we did rearrange the code with schilytools 2019-07-15.
We now have the needed #undef USE_ACL in diff.c as well.
Thanks to Dennis Clarke for reporting.
- star: Avoid a warning when a star -dump archive is unpacked on FreeBSD
or Linux with non-contiguous minor bits. The warning is not needed
since we only use SCHILY.dev in order to detect mount points but
not to compute the major/minor parts.
Release 2019-08-13:
- star: star -xdev -find typically works to exclude mounted files.
It still does not always do what is expected, e.g. in case that /proc
is in the tree of scanned files, where files deep in the new mounted
tree suddenly have the same FS ID as other filesystems, e.g. the file
/proc/<pid>/path/a.out.
In such cases, "star -find -xdev" is still recommended where the
mounted file exclusion is done inside libfind instead of being
done inside star.
Before, files on other filesystems have not been honored at all when
using "star -xdev -find ...".
Update provided by Michael Bäuerle via pkgsrc-wip.
Changelog
=========
Release 2018-11-22:
-libschily: resolvenpath() did not work as expected when some path names
do not exist. A stat() call that should check whether we already
reached the "/" directory caused a return (-1) even with
(flags & RSPF_EXIST) == 0
This bug caused star to classify more symlinks as dangerous than
needed.
- star: A typo in the function dolchmodat() has been fixed. The bug has been
introduced in July 2018 while adding support for very long path names.
- star: added a new timestamp to the star version.
- star: The man page now mentions incremental backups and restores in the
FEATURES section.
Release 2018-12-06:
- star: hole.c: A memory leak in in hole.c::put_sparse() has been fixed.
Thanks to Pavel Raiskup for reporting this coverity result.
- star: xheader.c: the macro scopy() no longer has a semicolon at the end.
Thanks to Pavel Raiskup for reporting this coverity result.
Release 2019-01-22:
- libstrar & star unicode.c: iconv() may return > 0 if there are
characters that could not be converted into an
identical meaning.
We therefore now check for ret != 0 instead of
ret == -1.
- star: added support for auto detection of "zstd" compressed archives.
- star: added a new option -zstd to support compression and uncompression
using the program "zstd".
- star: Recently, star did hang in the FIFO code on Solaris. This did
not happen on Solaris over 20 years before...
On Linux - on fast multi CPU machines - the probability that a
child process from fork() starts up before the parent is 1000x higher
than on Solaris, where 10 million tries were needed to reproduce the
same problem.
As a result, the FIFO in star on Linux could in rare cases (1 of.
~ 10000 tries) even finish the 1st read() from the input file before
the "tar"-process starts with e.g. command lines like "star -tv" or.
"star -x". Since star introduced auto-byte-order detection and
handling in 1985, star needs a special start up sequence to do that.
Star introduced the FIFO in the late 1980s and the machines from that
time did always restart the parent before the fork()ed child starts.
The new OS behavior thus caused a situation that was not forseeable
when the FIFO has been designed. This new OS behavior caused a
deadlock in aprox. 1 of 10000 star calls on Linux and 1 of 10000000
star calls on Solaris.
Star now waits when entering the FIFO fill-process until the.
FIFO get-process did start up before trying to wake up a waiting
get process.
- star: On Linux, in 1 of 1.5 million tries, star did die from SIGPIPE.
Note that this did never happen on Solaris.
Star now ignores SIGPIPE and it seems that this fixed the problem
since it did not happen again after that change with even 100 million
tries.
- star: The debug printing for the FIFO has been enhanced to print more
information from the FIFO control structure to make it easier to debug
problems like the ones mentioned above.
- star: There seems to be a problem in pipe handling in the Linux kernel.
It seems that in rare cases, the read(2) on a pipe returns 0 even though
the write side did write(2) one byte to the pipe just before calling
exit(). Unfortunately, this problem is hard to debug as it happens only
once every ~30 million tries. Our workaround is to behave as if the
expected byte could be read and star currently prints something like:
star: Erfolg. Sync pipe read error pid 8141 ret 0
star: Erfolg. Ib 0 Ob 1 e 0 p 1 g 0 chan 5.
star: Erfolg. Trying to work around Kernel Pipe botch.
before it continues. Since the star exit code in such a case is 0,
we assume that this is a correct workaround and this case thus may
be made completely silent in the future.
- star: an even less frequent FIFO problem (occurs once every 50 million
tries on fast multi CPU machines) has been identified. Star reports a
hard EOF on input even though the complete file with logical EOF has
been read and there is still input to process. In order to debug this
problem a debug message has been added to the code.
With this debug message, it turned out, that this problem happened
because a context switch occurred in the FIFO read process after it did
see an empty FIFO and later, after the process was resumed, the
following check for the FIFO_MEOF flag did see EOF. We now first check
for the FIFO_MEOF flag and later for the amount of data inside as the
FIFO as FIFO_MEOF is set after the FIFO content has been updated and
thus a context switch is no longer able to cause a wrong assumption
about the content of the FIFO.
If you still see this, please send a report.
- star: added support to print debug malloc statistics to better debug
memory problems in star.
- star: pathname.c:: free_pspace() now only frees the path buffer if it
is != NULL
- star: fixed a bug in the file create.c that caused star to incorrectly
grow the path buffer by 2 bytes for every archived file. This caused
star to constantly grow if a larger amount of files are archived and
eat up all memory available to 32 bit processes if the archived
filesystem is larger than approx. 1 TB.
- star: If the path name now cannot be handled because of low memory,
we print a warning that includes the text "out of memory".
- star: Now checking whether open of /dev/null failed while running a
compress pipe. This avoids a core dump on defective OS installations.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: props.c: Added a missing /* FALLTHROUGH */ comment..
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: create.c: Add more comment for the CPIO CRC format handler to
explain why the last instance if a series of hard links for a file
needs to archive the data.
- star: diff.c: added a filling fillbytes(&finfo, ...) to make sure that
ACL pointers are initialized.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: Several /* NOTREACHED */ comments have been added to tell
programs like coverity that after a NULL pointer check, there is no
continuation of the program
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: extract.c: A if (path->ps_path == '\0') has been corrected to
if (path->ps_path[0] == '\0') after a mktemp() call. This was a typo
introduced with the new support for extremely long path names.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: extract.c An initalization for a struct pathstore has been
moved to the front to verify that path.ps_path is always initialized.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: header.c: isgnumagic(&ptb->dbuf.t_vers) has been changed to
isgnumagic(ptb->ustar_dbuf.t_magic) as it is a "ustar" structure
that is going to be checked.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: some Cstyle changes
- bsh / Bourne Shell / star: the function hop_dirs() no longer checks
for p2 != NULL before calling *p2 = '/' as p2 has
been granted to be != NULL from a break with
strchr(p, '/') == NULL
Release 2019-02-18:
- star: another similar has been fixed similat to what has been fixed
already in the 2019-01-22 release:
An even less frequent FIFO problem (occurs once every 50 million
tries on fast multi CPU machines) has been identified. Star reports a
hard EOF on input even though the complete file with logical EOF has
been read and there is still input to process. In order to debug this
problem a debug message has been added to the code.
With this debug message, it turned out, that this problem happened
because a context switch occurred in the FIFO read process after it did
see an empty FIFO and later, after the process was resumed, the
following check for the FIFO_MEOF flag did see EOF. We now first check
for the FIFO_MEOF flag and later for the amount of data inside as the
FIFO as FIFO_MEOF is set after the FIFO content has been updated and
thus a context switch is no longer able to cause a wrong assumption
about the content of the FIFO.
We now did run 250 million tests without seeing another problem.
If you still see this, please send a report.
- star: Note that the debug output for this problem now has been
disabled. If you need to debug this, call:
smake clean COPTX=-DFIFO_EOF_DEBUG all
in the star directory.
- star: The message "Sync pipe read error" is no longer printed when
the FIFO background process dies instead of sending a final wakeup.
This is needed since there is a possibility for a context switch in
the foreground process that can make it later wait for a wakeup while
the background process misses to see the wait flag and just exits.
- star: In rare conditions (once every 2 million tries), a hang could.
occur with "star -c" if the tar process fills the FIFO and sets the
EOF flag and then calls wait() to wait for the FIFO tape output
process. This happens in case that the tape output did not see the
EOF flag because it has undergone a context switch after it checked
for the not yet existing EOF flag and before waiting for a wakeup
from the tar FIFO fill process.
Star now closes the sync pipes before calling wait() as this always
wakes up the waiting other side.
We did run another 300 million tests for this condition and did not
see any problem now.
- star: The version is now 1.6
Short overview for what changed since the last "stable" version:
- Support for "infinitely" long path names has been added.
- Support for comparing timestamps with nanosecond granularity
- -secure-links has been made the default when extracting
archived (except when doing an incremental restore).
- Added Support for NFSv4 ACLs on FreeBSD. Solaris has been
supported since 2013.
- Added Support to archive SELinix attributes.
- Allow to configure whether "star -fsync" is the default in
order to support filesystems that are slow with granted
transactions (like ZFS) or platforms that are genrally
slow with fsync() (like Linux).
- Full UNICODE support has been added for tar headers.
- Support for -zstd compression has been added.
- Some rare FIFO problems have been fixed.
Note that we did recently run more than a billion tests to
verify the FIFO after we identified a method to trigger the
problem on Linux.
Release 2019-03-11:
- star: Support for base-256 numbers in timestams and uid/gid has been
added. This has been planned in the 1990s already, when star invented
the base-256 coding, but it has been forgotten in favor of the
POSIX.1-2001 enhanded archive headers. Now it seems that GNU tar.
that copied the format from star uses it for timestamps and uid/gid
and we need to implement it in order to get archive compatibility.
Thanks to Michal Górny (mgorny@gentoo.org) for detecting the missing
feature.
- star: The t_rdev field in the old star header now may use base-256
as well.
- star: The function stoli() added a new parameter "fieldwidth" that
allows to configure when a "unterminated octal number" warning is
printed. This is needed since this function is used for 8 byte and
for 12 byte fields.
- star: star did print archives with illegal 32 byte user/group.
names (where the nul terminator is missing) "correctly", when in.
list mode but it used only the first 31 bytes when extractig.
such archives
- star: a new function istarnumber() is used to do better heuristics on
what a valid TAR archive is. We have some special handling to work.
around the non-compliance of GNU tar in some known cases. If you
discover other GNU tar archives that are not detected as TAR archive,
please report them to help to make th eheuristics better.
The background is to make star better in detecting fool archives.
- star: The directory testscripts added new files:
testscripts/not_a_tar_file1 and testscripts/not_a_tar_file3
with correct checksums that fool tar implementations that use too
few heuristics to identify tar archives.
- star: fixed a bug in the FIFO related to extracting multi-volume
archives. The bug was introduced with release 2019-02-18 and the
effect was that the FIFO complained at the end of the last volume.
- star/libschily: Added new error checking codes:
"ID"<-->allows to control error behaviour with range errors in uid_t
and gid_t values.
"TIME"<>allows to control error behaviour with range errors in time_t
- star: Creating multi volume archives without using the FIFO did dump
core. We thus no longer set mp->chreel = TRUE; when the FIFO has.
been disabled. The related bug has been introduced in January 2012.
- star: Creating multi volume archives with a very small volume size
could cause a hang at the end as the function startvol() did not
check whether the TAR process did already decide to exit while
waiting for the TAR process to calm down (stop) before writing the
next multi volume header. We no longer wait in this case.
- star: exprstats() now calls fifo_exit(ret) in order to avoid a
FIFO Sync pipe read error message in case that star was terminated
with an error.
- star: Since we added better Unicode support in May 2018, star did
dump core when a multi volume header with POSIX.1-2001 extensions
was written in multi volume create mode. We now check for NULL
pointers before we call nameascii() to decide whether the file.
name needs a UTF-8 translation.
- star: Creating multi volume archives without POSIX.1-2001 support
no longer sets POSIX.1-2001 extension flags for the volume header.
- star: The flag XF_NOTIME now works when creating POSIX.1-2001
extended headers and thus the 'x'-header with time stamps for the
volume header tar header is no longer created. This avoids
to write atime=1 for the volume number 1 since we encode the
volume number in the otherwise useless atime of the volume header
when in POSIX.1-1988 TAR mode.
- star: the star.1 man page now mentions that the first tar program
appeared in 1979 (3 years before star has been started as a project).
- star: the star.4 man page now has a "SEE ALSO", a HISTORY and
a AUTHOR section.
- star: the star.4 man page now has a MULTI VOLUME ARCHIVE HANDLING
section.
- star: the star.4 man page added a new "BASIC TAR STRUCTURE" section.
- star: The ACL reference test archives (formerly available from e.g.:
http://sf.net/projects/s-tar/files/alpha/) have been added
to the directory star/testscripts/. The files.
acl-test.tar.gz
acl-test2.tar.gz
acl-test3.tar.gz
acl-test4.tar.gz
acl-test5.tar.gz
contain ACLs that use the obsolete method from a POSIX proposal
from around 1993 that was withdrawn in 1997 and never has become
part of a standard. This method has been implemented in 1993 for
UFS on Solaris.
GNU tar claims to support this format but really does not support
it at all. GNU tar fails to extract the reference tar archives from
above and it fails to create a compliant tar archive in create mode.
It is strange to see that GNU tar never has been tested against the
reference archives that have been created in collaboration with
SuSE in 2001 already.
The files
acl-nfsv4-test.tar.gz
acl-nfsv4-test2.tar.gz
acl-nfsv4-test3.tar.gz
acl-nfsv4-test4.tar.gz
acl-nfsv4-test5.tar.gz
contain ACLs that have become part of the NFSv4 standard and that.
are also used on NTFS and ZFS. This format is completely unsupported
by GNU tar.
- star TODO: create unit tests in order to avoid future problems
with multi volume archives similar to the problems we recently
fixed.
- star: Updated version 1.6 (not yet published in separate tarball)
Short overview for what changed since the last "stable" version:
- Support for "infinitely" long path names has been added.
- Support for base-256 numbers in timestams and uid/gid
has been added. This has been planned in the 1990s already,
when star invented the base-256 coding, but it has been
forgotten in favor of the POSIX.1-2001 enhanded archive
headers.
- Support for comparing timestamps with nanosecond granularity
- -secure-links has been made the default when extracting
archived (except when doing an incremental restore).
- Added Support for NFSv4 ACLs on FreeBSD. Solaris has been
supported since 2013.
- Added Support to archive SELinix attributes.
- Allow to configure whether "star -fsync" is the default in
order to support filesystems that are slow with granted
transactions (like ZFS) or platforms that are genrally
slow with fsync() (like Linux).
- Full UNICODE support has been added for tar headers.
- Support for -zstd compression has been added.
- Some rare FIFO problems have been fixed.
Note that we did recently run more than a billion tests to
verify the FIFO after we identified a method to trigger the
problem on Linux.
Problems found with existing distfile for eagle:
distfiles/bicom101.zip
distfiles/szip-2.1nb3/szip-2.1.tar.gz
distfiles/xmill-0.9.1.tar.gz
No changes made to these distinfo files.
Otherwise, existing SHA1 digests verified and found to be the same on
the machine holding the existing distfiles (morden). All existing
SHA1 digests retained for now as an audit trail.
