This library is for working with ".tar" archive files. It can read and
write a range of common variations of archive format including V7,
POSIX USTAR and GNU formats.
It provides support for packing and unpacking portable archives. This
makes it suitable for distribution but not backup because details like
file ownership and exact permissions are not preserved.
It also provides features for random access to archive content using
an index.
pkglint --only "https instead of http" -r -F
With manual adjustments afterwards since pkglint 19.4.4 fixed a few
indentations in unrelated lines.
This mainly affects projects hosted at SourceForce, as well as
freedesktop.org, CTAN and GNU.
Update ruby-mixlib-archive to 1.0.5.
1.0.5 (2019-12-30)
* Remove Travis and update Github templates #40 (tas50)
* Chefstyle fixes + test on Windows in Buildkite #43 (tas50)
* Test on Ruby 2.7rc and other testing updates #47 (tas50)
* Substitute require for require_relative #48 (tas50)
Changes from 1.0.2 to 1.0.4 are unknown.
Update ruby-ffi-libarchive to 1.0.0.
* Update HOMEPAGE and LICENSE.
* Proper COMMENT.
* No need to use gmake.
* add "USE_LANGUAGES= # none".
1.0.0 (2019-12-28)
* Chefstyle fixes to get the build green again #25 (tas50)
* Add support for custom read functions #27 (jatoben)
* Add Ruby 2.7rc testing and cache gem installs in Buildkite #28 (tas50)
The zip-archive library provides functions for creating, modifying,
and extracting files from zip archives. The zip archive format is
documented in http://www.pkware.com/documents/casestudies/APPNOTE.TXT.
Certain simplifying assumptions are made about the zip archives: in
particular, there is no support for strong encryption, zip files that
span multiple disks, ZIP64, OS-specific file attributes, or
compression methods other than Deflate. However, the library should be
able to read the most common zip archives, and the archives it
produces should be readable by all standard unzip programs.
Change log:
* Translations update
* fr-command-rpm: Use rpm2cpio binary shipped with rpm package
* add source rpm to supported mime-types
* use cpio if no binary is found
* add a configure environment variable to select the cpio path
* help: pull updated Russian translation to remove broken one
* Fix localized pause UI logic
Based on Schily Tools Release 2019-12-05.
AN-2019-11-11:
- libschily: the new functions wcastoi wcastol wcastolb that have been
added with the previous release have been forgotten in the linker map
file.
- star: When using star -copy, the default is now -no-secure-links.
This is what users like since they are copying their own data that
per definition cannot be untrusted external data.
- star: star -diff is now "more smooth" to use, when the archive type
is GNU TAR and the "atime" or "ctime" properties are not present
for a specific file. Before, Jan 1 1970 was the assumed time in
the archive.
- star: star -diff now prints a diff statistics summary.
- star: New version date
AN-2019-12-05:
- include/schily/stdio.h: We no longer define getc_unlocked() on Solaris
in case that it is defined already. This was needed since Oracle
Solaris 11.4 compiles 64 bit by default and made stdio less opaque
than before - this result in the same bahavior as for 32 bit programs.
This means, we no longer need to use tricks to get fast access to
stdio.
Thanks to Rolf Dietze for reporting.
- libschily: the function getargs() and similar (from the option parser
group) fixed a bug that prevented the usability of the official method
to include the char '+' inside option names by quoting the character
in the option name used in the format string.
This bug has been fixed in order to be able to support "star -dump+ ..."
- libshedit: because of the changes in Oracle Solaris 11.4, we need to
#undef FAST_GETC_PUTC in order to avoid to use getc_unlocked().
Thanks to Rolf Dietze for reporting.
- libfind: A new function walksname() has been added to set up the
internals in "struct WALK" for external path names. This feature is
needed to support "star -c list=names -find ...".
- libfind: The functions fetchdir()/dfetchdir(), sortdir(), cmpdir() now
use size_t instead of int for the entry number count and the allocated
size.
This breaks binary compatibility in 64 bit mode for those users only,
that use the interfaces fetchdir()/dfetchdir(), sortdir(), cmpdir().
The only application that currently uses these interfaces is "star".
Simple users of libfind and programs compiled as 32 bit programs
are not affected by the change.
Warning: packetizers need to take care of the changed name of the
shared library file for libfind.
- libfind: Because of the binary incompatible change, we incremented
the major library version number from 3 to 4.
WARNING: This affects only distros that ship 64 bit versions of star
using a shared version of libfind.
Since Solaris defaults to 32 bit programs, this is expected to only
affect Linux distros.
- libstreamar: unicode.c has been changed to make sure that is works even
in case that the "len" parameter is an unsigned.
- libstreamar: to_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- libstreamar: from_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: The new-volume-script= script is now called at the end of
_every tape_, even at the last one. This now allows to have a script
that automatically renames a standard archive name into numbered
archive names when a tree should be splitted into chunks, e.g. with
this command line:
star -c f=/tmp/v.tar tsize=1G new-volume-script='sh -c \
"mv /tmp/v.tar /tmp/v$(($1-1)).tar" nv' .
you get partial archives named /tmp/v1.tar ... /tmp/vn.tar
Note that this would not work in case that "sh" is the historical
Bourne Shell, e.g. as seen on Solaris 10. This is because
"shell arithmetics" is a feature that was not present in the historical
Bourne Shell.
- star: The man page now mentions that "star cli=tar ..." works as an
alias to "star cli=suntar ...".
- star: The man page now mentions that "star cli=gnutar ..." does not
impement the bugs from the GNU option parser.
- star: Is "star cli=xxx ..." is used, then star now calls
set_progname(xxx) and thus prints error messages labelled with the
command line interface name.
- star: In "pax" command line mode, star now uses -no-fsync as default
in order to behave the same way as the closed source Solaris pax
does.
- star: Star now allows to combine the option list=names with -find
This is useful, whenever a list of filenames is longer than ARG_MAX and
thus cannot be used in the star command line after -find.
Here is an example on how to use the new feature in a mercurial.
based development environment:
(hg status -a; hg status -m; hg status -c) | sed -e 's/^. //' > .hgfiles
star -c list=.hgfiles -find -mtime -100 > /tmp/new.tar
The first command puts all files under control of mercurial into
thw file ".hgfiles" , while omitting the compile results from the
file ".hgfiles".
The second command takes that list and puts only those files into
the archive that have been modified during the past 100 days.
- star: A new option "-dump+" allows to have the "SCHILY.volhdr.hostname"
meta data field filled out even when not making dumps with an
associated dump level. This helps to remember the origin of tar
archives.
- star: unicode.c has been changed to use the same comment as unicode.c
in libstreamar.
- star: unicode.c has been changed to make sure that is works even.
in case that the "len" parameter is an unsigned.
- star: to_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: from_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: due to the incompatible interface change in libfind (see
above), several changes have been introduced in order to use size_t
instead of int for fetchdir()/dfetchdir(), sortdir(), cmpdir().
This finally permits star to be able to make use of the current POSIX
maximum meta data size for so called "pax" archives (TAR archives
with POSIX.1-2001 meta data enhancements derived from the Solaris 7
concept for tar from 1997). Previous versions of star have been
limited to data no longer than 2 GB (even though the star main code
supports 8 GB file size), this version now supports 8 GB for the.
sum of all meta data for a single file in case a 64 bit binary from
star is used.
- star: the fifo now has a new variant of te function runfifo() that
is used by the new fifo(1) standalone program. See below.
- star: The README now mentions the new "star cli= ..." method and
the fact that star deals with infinite path name length.
0.6.2.1
* Add support for GHC 8.8 / base-4.13
0.6.2
* New cabal flag 'pkg-config' for discovering 'zlib` via pkg-config(1) (#16)
* Use CApiFFI where available for cross-compile friendliness (#14)
* Change the window bits range from 8..15 to 9..15 (#11)
0.6.1.2
* Fix a segfault when reading the stream multithreaded, #7
* New experimental cabal flag 'non-blocking-ffi' for 'safe' ffi calls
0.6.1.1
* Fixed building with GHC 7.0 and 7.2
0.6.0.2
* Fixed building with GHC 7.0 and 7.2
0.6.1.0
* Support for concatenated gzip files (multiple back-to-back streams)
0.6.0.1
* Fixed building with older GHC
* Fixed warnings with new GHC
* Fixed building on Windows
* Fixed testsuite
0.6.0.0
* New incremental interface for compression and decompression
* Provide access to unconsumed trailing data
* Simplified structured error type, and instance of Exception
* Updated bundled zlib C code to 1.2.8 (used on Windows)
* Fixed memory leak of zlib z_stream C structure
* More derivied instances (Eq, Show, Typeable, Generic)
0.5.4.2
* Builds with GHC 7.10
Changes 0.13.0:
* ``pytest-xdist`` ``pytest`` extension is now installed so tests can be
run in parallel.
* CI now builds ``manylinux2010`` and ``manylinux2014`` binary wheels
instead of a mix of ``manylinux2010`` and ``manylinux1``.
* Official support for Python 3.8 has been added.
* Bundled zstandard library upgraded from 1.4.3 to 1.4.4.
* Python code has been reformatted with black.
At least ZSTD_FRAMEHEADERSIZE_MIN and ZSTD_FRAMEHEADERSIZE_PREFIX
macros in 1.4.4 needs a `format' argument but py-zstandard is not
adapted to 1.4.4 yet.
PKGREVISION++
Brotli is a generic-purpose lossless compression algorithm that compresses data
using a combination of a modern variant of the LZ77 algorithm, Huffman coding
and 2nd order context modeling, with a compression ratio comparable to the best
currently available general-purpose compression methods. It is similar in speed
with deflate but offers more dense compression.
This package contains Python bindings for Brotli.
2.9:
Changes to the library:
optimized the ArchiveEntry class
added support for the zstd format
fixed the top-level __all__ list (7b97d8b)
updated the code to support Python 3.8
improved the add_file_from_memory method
lowered the level of log messages from warning to info
Tests:
dropped testing with Python 3.4, added 3.7 and 3.8
fixed unicode handling in entry tests
Upstream changes:
0.86 Thu Dec 5 20:53:07 GMT 2019
* Fix RT#131072 which was due to tar.exe and CRLF line endings
0.84 Wed Dec 4 21:20:04 GMT 2019
* Fix RT#119084 - won't compile on MSWin32 with taint
0.82 Fri Nov 22 19:09:35 GMT 2019
* Correctly compare version string for Archive::Tar (#7)
* Remove duplicate entry from the MANIFEST file. (#6)
* Treat minix the same as NetBSD
innoextract 1.8 (2019-09-15)
- Added support for Inno Setup 6.0.0 installers
- Added support for pre-release Inno Setup 5.6.2 installers used by GOG
- Added support for two modified Inno Setup 5.5.7 variants
- Added support for Inno Setup 1.3.0 to 1.3.23
- Added support for My Inno Setup Extensions installers older than 3.0.6.1
- Added support for modified Inno Setup variants using an alternative setup loader magic
- Added support for using boost_{zlib,bzip2} when statically linking Boost
- Added support for automatically reading external setup.0 files
- Encoding for non-Unicode installers is now determined from the languages supported by the installer, overridable using the --codepage option
- Implemented parsing of GOG Galaxy architecture constraints
- The architecture-specific suffixes @32bit and @64bit are now used to disambiguate colliding files
- Fixed extracting files from slices larger than 2 GiB with 32-bit builds
- Fixed output path for files with absolute paths (canonicalization now strips all unsafe characters)
- Fixed output directory being created even when not extracting files
- Fixed a hang when using the --language option
- Improved checksum verification for files reconstructed from GOG Galaxy file parts
- Changed header parsing to select the first version without warnings and failing that the first without errors
- Changed filesystem and output encoding to WTF-8 (extended UTF-8) to represent broken UTF-16 data
Based on schilytools release from 2019-10-25.
Changelog
=========
- libstreamer: Added new keywords "fsdevmajor" and "fsdevminor" as
there are too many incompatible st_dev binary formats.
These keywords are not yet created, but they are understood.
This permits to switch to the new format (fade out "dev") later.
- star: Added new keywords "SCHILY.fsdevmajor" and "SCHILY.fsdevminor"
as there are too many incompatible st_dev binary formats.
These keywords are not yet created, but they are understood.
This permits to switch to the new format (fade out "SCHILY.dev") later.
- star: The amount of shared mamory used by default has been reduced to
1 MB on Ultrix. These are old small machines, that do not support 8 MB.
Zstandard v1.4.4
This release includes some major performance improvements and new CLI features, which make it a recommended upgrade.
perf: Improved decompression speed, by > 10%
perf: Better compression speed when re-using a context
perf: Fix compression ratio when compressing large files with small dictionary
perf: zstd reference encoder can generate RLE blocks
perf: minor generic speed optimization
api: new ability to extract sequences from the parser for analysis
api: fixed decoding of magic-less frames
api: fixed ZSTD_initCStream_advanced() performance with fast modes, reported by @QrczakMK
cli: Named pipes support
cli: short tar's extension support
cli: command --output-dir-flat=DIE , generates target files into requested directory
cli: commands --stream-size=# and --size-hint=#
cli: command --exclude-compressed
cli: faster -t test mode
cli: improved some error messages
cli: fix rare deadlock condition within dictionary builder
build: single-file decoder with emscripten compilation script
build: fixed zlibWrapper compilation on Visual Studio, reported by @bluenlive
build: fixed deprecation warning for certain gcc version, reported by @jasonma163
build: fix compilation on old gcc versions
build: improved installation directories for cmake script, by Dmitri Shubin
pack: modified pkgconfig, for better integration into openwrt, requested by @neheb
misc: Improved documentation : ZSTD_CLEVEL, DYNAMIC_BMI2, ZSTD_CDict, function deprecation, zstd format
misc: fixed educational decoder : accept larger literals section, and removed UNALIGNED() macro
(from: HomePage)
0.37 (2019/08/24):
Fixed issue 14 - https://github.com/glasswalk3r/archive-tar-wrapper-perl/issues/14
0.36 (2019/04/11):
Bug fixes since tests on Windows were failing after latests changes.
Refactored code due unexpected way bsdtar on Windows and tar on OpenBSD behave.
Refactored the tests due the bugs found.
Small fixes on Pod, specially the section for MS Windows users.
Added the missing tests to the MANIFEST.
0.35 (2019/04/05):
Added _rem_dots method, a improved way to remove '.' and '..' from readdir(), supposed to be faster than previous implementation with sort() and shift().
Added two more tests to increase testing coverage.
Fixed a small bug regarding Pod, that now covers 100% of the code.
Small fixes to documentation.
0.34 (2019/03/22):
Increased test coverage.
Refactored code to make it easier for testing.
Disabled performance-small.t for now.
Refactored bzip2 tests.
Adding META.yml "provides" from Makefile.PL
Covered methods that didn't have Pod.
Refactored all Pod following best practices.
Refactored a single method to be "private".
Slightly changed new() to enable unit testing for OS specific configurations.
zipstream.py is a zip archive generator based on python 3.3's
zipfile.py. It was created to generate a zip file generator for
streaming (ie web apps). This is beneficial for when you want to
provide a downloadable archive of a large collection of regular files,
which would be infeasible to generate the archive prior to downloading
or of a very large file that you do not want to store entirely on disk
or on memory.
The archive is generated as an iterator of strings, which, when
joined, form the zip archive.
Based on Release 2019-10-07.
Changelog
=========
- configure: Some shells report a syntax error with "< file (cmd)"
and need the redirection statement to be *after* the command. Our
changes to support the V7 shell by adding round braces caused ash
variants like "dash" to fail.
Thanks to Harald van Dijk for reporting
- cont/cc-config.sh: canged some :>some-file statements into
(:)>some-file. they have meen missed when introducing work arounds
for the V7 Shell on Ultrix that does not support I/O redirection
for builtin commands.
Thanks to Robert Clausecker for reporting
- libschily/resolvepath.c: resolving a symlink that points to another
symlink that points to itself, caused a coredump as a result from an
endless recursion.
We now detect this situation and abort the check before the endless
recursion causes a stack overflow. A symlink that directly loops
is immediately stopped. A longer symlink loop chain over more than one
symlink can only be detected by the recursion nesting level and is
aborted after a nesting level of 1024 has been reached. This works
under the assumption that the minimum stack size is more than
1024 * PATH_MAX and that there is no useful directory path with more
than 1024 symlinks in the path.
----> This problem affected star and SCCS.
Thanks to Philipp Wellner for reporting
- star: Added a hint to the man page that helps to find pkglist= as a.
similar option to list=
- star: The new method to avoid extracting symlinks that point outside
the star working directory that has been introduced in October 2018
could cause a core dump if a symlink is checked that points to
another aready existing symlink that points to itself. This was caused
by a problem in libschily/resolvepath.c, see above.
Thanks to Philipp Wellner for reporting
- star: The option -no-secure-links now may be configured as a global
default via the tag STAR_SECURE_LINKS= in the file /etc/default/star
and as a private default via an environment of the same name.
If the value for this tag is 'n' or 'N', -no-secure-links is made the
default, any other value sets the option -secure-links as the default.
This may be useful for sysadmins that frequently use star to copy
installation specific files, but it is risky in case that alien TAR
archives are imported. The good news is that this permits users to
switch to the old star behavior where no checks for risky links
existed.
Thanks to Dennis Clarke for reporting
- star: A new enviroment STAR_NOHINT has been introduced to supress
hint messages that are otherwise seen in case STAR_SECURE_LINKS or
STAR_FSYNC is in the environment or in /etc/default/star
- star: New version date
Update archivers/ruby-minitar to 0.9.
## 0.9 / 2019-09-04
* jtappa added the ability to skip fsync with a new option to Minitar.unpack
and Minitar::Input#extract_entry. Provide `:fsync => false` as the last
parameter to enable. Merged from a modified version of PR [#37][].
## 0.8 / 2019-01-05
* inkstak resolved an issue introduced in the fix for [#31][] by allowing
spaces to be considered valid characters in strict octal handling. Octal
conversion ignores leading spaces. Merged from a slightly modified version
of PR [#35][].
* dearblue contributed PR [#32][] providing an explicit call to #bytesize for
strings that include multibyte characters. The PR has been modified to be
compatible with older versions of Ruby and extend tests.
* Akinori MUSHA (knu) contributed PR [#36][] that treats certain badly
encoded regular files (with names ending in `/`) as if they were
directories on decode.
## 0.7 / 2018-02-19
* Fixed issue [#28][] with a modified version of PR [#29][] covering the
security policy and position for Minitar. Thanks so much to ooooooo\_q for
the report and an initial patch. Additional information was added as
[#30][].
* dearblue contributed PR [#33][] providing a fix for Minitar::Reader when
the IO-like object does not have a `#pos` method.
* Kevin McDermott contributed PR [#34][] so that an InvalidTarStream is
raised if the tar header is not valid, preventing incorrect streaming of
files from a non-tarfile. This is a minor breaking change, so the version
has been bumped accordingly.
* Kazuyoshi Kato contributed PR [#26][] providing support for the GNU tar
long filename extension.
* Addressed a potential DOS with negative size fields in tar headers
([#31][]). This has been handled in two ways: the size field in a tar
header is interpreted as a strict octal value and the Minitar reader will
raise an InvalidTarStream if the size ends up being negative anyway.
ADVANCECOMP VERSION 2.1 2018/02
===============================
* Support ZIPs with data descriptor signature.
* Fixed a crash condition with invalid ZIP data.
ADVANCECOMP VERSION 2.0 2017/06
===============================
* Added support for reading MNG files with depth of 1, 2, and 4 bits.
* Added 64 bits binary for Windows.
* Updated to libdeflate 29-May-2017.
From https://github.com/ebiggers/libdeflate
at commit 1726e9e87fb6f98682dfdea2356d5ee58881fe7b.
Switch to latest distfile.
Update PLIST.
Replaced MESSAGE with share/doc/star/INSTALL.pkgsrc.
Changelog
=========
Release 2019-03-29:
- libschily: the lutimens() emulation no longer returns ENOSYS in case
that the lstat() call fails, since this error code may e.g. be
ENAMETOOLONG and used as an important indicator for long path name
handling.
This bug caused incorrect behavior when star extracted long pathnames
on a platform without utimensat().
- star: The code restructuring to openat() from Summer 2018 did not
only cause a noticeable speed up even when not using -find, it at
the same time resulted in a wrong error message when a file type
was seen that is not archivable (e.g. a socket while using the USTAR
format). The correct message was something like "Unsupported filetype",
while the bug caused a "file not found " message. This has been fixed
by adding a new parameter "fd" to the function stat_to_info().
- star: header.c::get_xhtype() we did forgot to initialize:
finfo.f_devminorbits
and
finfo.f_xflags
This is now done.
- star: The USTAR format now is able to create base-256 values in
the field "t_devmajor". "t_devminor" did already support base-256.
- star: The GNU tar format now is able to create base-256 values in
the fields "t_devmajor" and "t_devminor".
- star: The man page star.4 now mentions which fileds may have
base-256 numbers.
- star: The man page star.4 now longer contains the doubled "field field"
in the explanation of the added UID/GID number for ACLs.
- star: The man page star.4 now has a better description for the additional
numeric fields in the ACL entries to prevent GNU tar from continuing
with it's incompatible ACL implementation.
- star: New ACL reference archives for the ultra compact format, that has
been defined together with the libachive people, have been added to the
directory testscripts/:
acl-nfsv4-compact-test.tar.gz
acl-nfsv4-compact-test2.tar.gz
acl-nfsv4-compact-test3.tar.gz
acl-nfsv4-compact-test4.tar.gz
acl-nfsv4-compact-test5.tar.gz
- star: Linux now uses the new <linux/fs.h> instead of <ext2/ext2_fs.h>
to retrieve file flags.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for the following new BSD Flags:
compressed hidden offline rdonly reparse sparse system
has been added.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for the following new Linux flags:
dirsync nocow notail projinherit topdir
has been added.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for reading non-comliant libarchive tarballs that use.
"securedeletion" or "journal"
instead of the documented text has been added. This was used by
libarchive until March 20, when libarchive has been fixed. We introduce
the above names for compatibility with old TARs created by libarchive.
- star: When an "old star" archive is read and this archive is from a system
like FreeBSD with non-continous minor bits, star no longer warns unless
there is a device file in the archive.
- star: unit tests: the scripts now contain
LC_ALL=C export LC_ALL
instead of just LC_ALL=C to make the environment exported.
- star/libstrar: The case where iconv() returns -1 and sets errno to E2BIG
is now handled correctly.
- star: The testscripts/ directory now contains the tar test archives
from the portability tests from Michal Gorny. Check
http://cdrtools.sourceforge.net/private/portability-of-tar-features.html
for an updated (to match star-1.6) variant of the results from the portability
tests from Michal Gorny.
- star: older versions of star did not print the messages:
"WARNING: Archive is 'xxx' compressed, trying to use the -xxx option.\n"
in case that the option -print-artype was in use. Later versions that
added support for more than "gzip" and "bzip2" forgot to add the same
exception for the new compression methods. We now ommit this message
for all compression types.
- star: Support for the "lzma" compression has been added.
- star: Support for the "freeze2" compression has been added.
- star: The compression method name list did not include the text "zstd".
We added the missing text.
- star: The hint messages that are printed when a compressed input archive
is not seekable did forgot to mention "lzip" and "zstd". We added the
forgotten messages.
- star: New unit tests check whether star is able to auto-detect and auto
decompress various compression formats.
- star: The unit tests now include the portability tests from Michal Gorny
- A new option cli=name (must be argv[1]) allows to select all
supported command line interfaces (star, suntar, gnutar,
pax, cpio) when called as star.
This is needed to be able to test all command line interfaces from
out unit tests since star is not installed in this case and a selection
from argv[0] would not work.
- star: The version date is now "2019/03/20"
Release 2019-04-29:
- libfind/star: the verbose listing code has ben restructured to have
the file permissions in the same string as the file type.
This is needed to implement the POSIX pax listformat interpreter
format %M in future.
- libfind/star: the verbose listing no longer prints "l" for mandatory
record locking for non-directory type file, but rather only
for plain files.
- star: The "gnutar" emulation now prints the --help output to stdout as
GNU tar does.
- star: The gnutar.1 man page now mentions that the GNU tar
options -g / -G did never work and it thus makes no sense to
implement them.
- star: very outdated code in list.c has been removed.
- star: Added new unit tests for incremental backups and restores.
This in special include tests that always fail with GNU tar
as GNU tar is not usable and never was usable for incremental
restores in case that the differences are more than trivial.
- star: new version date 2019-04-01
Release 2019-06-13:
- star: fixed a bug in the FIFO code that mainly happened on Linux (with
a 1000x higher probability than it happens on Solaris). The bug was.
caused, as a check for a flag has been done twice instead of only once
where it could change it's value between both locations. As a.
result, star reported "star: Implementation botch: with FIFO_MEOF"
as the tar side of the FIFO did sometimes not wait for the FIFO_IWAIT
state when called as "star -multivolume -tv f=... f=... ...
For this reason, star incorrectly got a wakeup at the wrong location.
The bug appeared in case that star -x/-t -multivol f=.. f=.. ....
has been called with very small tar archives.
Thanks to Heiko Eissfeldt for reporting.
- star: The FIFO code renamed the "flag" member of the "m_head" structure
to "gflag" for better readability.
- star: The debugging code in the FIFO has been enhanced to print the
names of the flag bits in addition to the hex values.
- star: A deadlock situation that happens once every 500000 tries on
Linux with multi-volume archives has been fixed.
The problem was caused by a complex condition where the get side
of the FIFO needs to check the EOF FIFO flag and the amount of data
available in the FIFO and then decide whether to wait for a wakeup or
not.
Since the EOF flag needs to be checked first, a context switch in
the get side of the FIFO could allow the put side to set the EOF flag
before the get side did check the fill ratio of the FIFO. This
caused both the get side and the put side to wait for a wakeup.
The new code introduced a new variable mp->mayoblock that is set by
the get side before checking for EOF. This new flags allows the
put side to know that the get side is just in a critical situation
and lets the put side wait until mp->mayoblock is no longer set,
which signals a stable state in the get side. This permits to
avoid the deadlock.
- star: Note that the FIFO has been initially written as a lock free
design in the late 1980's. This is to allow high portability to even
older UNIX versions. The star FIFO works on all UNIX variants that
support pipes and shared memory, which is e.g. the case for
SunOS-4.0 from 1988. At the time the FIFO has been designed, the
target OS did not support multi-CPU systems and problems in the
FIFO first appeared with massively faster multi-CPU systems around
y2000. The recently detected problems all have been triggered by a
different context switch behavior on Linux, even though they could
have appeared on any OS in case that many million tries are
attempted.
- star: bumped version to 1.6.1
Release 2019-07-15:
- star: the compress unit test no longer fails on Cygwin because the
gzip binary is compiled incorrectly and does not support LZW.
The related test is now skipped.
Thanks to Heiko Eissfeldt for reporting
- star: some new unit tests failed if the schilytools source tree has
been installed in a directory with spaces in it's name.
Thanks to Heiko Eissfeldt for reporting
- star: The unit tests for incremental backups include archives that
include userid/groupid and username/groupname that may not be
restorable on a different computer. We now ignore these ID meta.
data when comparing the results.
Thanks to Heiko Eissfeldt for reporting
- star: When comparing nanoseconds in time stamps, star now has a mode
that treats time stamps as equal in case that tv_nsec % 100 == 0
and the rest of the nano seconds is equal. This is needd on Cygwin
since NTFS counts in 1/10 microseconds since Januar 1 1601.
This applies to both star -diff and the "newer" check while extracting
files.
Thanks to Heiko Eissfeldt for reporting
- star: a new option diffopts=dnlink has been implemented to support
filesystems that do not follow the historic UNIX model for hard links
on directories.
Since Cygwin usually has a linkcount of 1 on directories, you need
to use "star -diff diffopts=!dnlink ..." if you like to diff the
meta data from a historical UNIX filesystem.
Thanks to Heiko Eissfeldt for reporting
- star: The incremental backup/restore tests now use.
"star -diff diffopts=!dnlink ..." to make them work on Cygwin.
Thanks to Heiko Eissfeldt for reporting
- star: star -c -H exustar -acl -xattr-linux .
did cause file not found messages from the attempt to archive the
Linux xattrs. This was caused by a change from July 2018 when trying to
optimize directory access in (non-find) create mode. The call to read
the Linux xattrs has now been moved to the location where in former
times the ACL code has already been moved.
The move is needed because there is no ACL/XATTR related function
that is similar to openat().
- star: A similar problem with get_xattr() did exist with star -diff
- star: star -diff did not compare ACLs since getinfo() no longer
includes a call to get_acl(). We now call get_xattr() and get_acl()
in diff.c
- star: The Solaris ACL interface now implelements lacl() / lacl_get() /
lacl_set() to support very long pathnames with ACLs.
- star: The Linux ACL interface now implelements lacl_get_file() /
lacl_set_file() to support very long pathnames with ACLs on platforms
that implement the withdrawn POSIX ACL draft.
- star: The Linux xattr interface now implelements.
llgetxattr() / llsetxattr() / lllistxattr() to support very long
pathnames with Linux xattrs.
- star: New version date
Release 2019-07-22:
- star: "pax -pe" no longer sets the variable "doxattr" as this caused
an error message:
"NFSv4 extended attribute files are not yet supported.\n"
that caused the OpenSolaris-ON "nightly" compilation to abort as a
result of that error and exit code != 0.
Release 2019-08-13:
- libschily: Various functions and *at() emulation functions call stat()
even though the caller does not know about that call.
This could cause a missbehavior in case that a file returns EOVERFLOW
with a normal stat(). We now compile these functions in unconditional
large file mode to overcome that problem.
The affected files are:
diropen.c lutimens.c findinpath.c linkat.c mkdirs.c searchinpath.c
resolvepath.c lchmod.c renameat.c.
Note that this problem affected star(1) on platforms that do not
fully implement all *at() interfaces, since star started to support
really long path names in July 2018.
- star: Star did not compile on platforms without ACLs anymore since
we did rearrange the code with schilytools 2019-07-15.
We now have the needed #undef USE_ACL in diff.c as well.
Thanks to Dennis Clarke for reporting.
- star: Avoid a warning when a star -dump archive is unpacked on FreeBSD
or Linux with non-contiguous minor bits. The warning is not needed
since we only use SCHILY.dev in order to detect mount points but
not to compute the major/minor parts.
Release 2019-08-13:
- star: star -xdev -find typically works to exclude mounted files.
It still does not always do what is expected, e.g. in case that /proc
is in the tree of scanned files, where files deep in the new mounted
tree suddenly have the same FS ID as other filesystems, e.g. the file
/proc/<pid>/path/a.out.
In such cases, "star -find -xdev" is still recommended where the
mounted file exclusion is done inside libfind instead of being
done inside star.
Before, files on other filesystems have not been honored at all when
using "star -xdev -find ...".
- improvements for Android APK and JAR archives
- better support for non-recursive list and extract
- tar --exclude-vcs support
- fixes for file attributes and flags handling
- zipx support
- rar 5.0 reader
0.12.0:
Backwards Compatibility Notes
* Support for Python 3.4 has been dropped since Python 3.4 is no longer
a supported Python version upstream. (But it will likely continue to
work until Python 2.7 support is dropped and we port to Python 3.5+
APIs.)
Bug Fixes
* Fix ``ZstdDecompressor.__init__`` on 64-bit big-endian systems.
* Fix memory leak in ``ZstdDecompressionReader.seek()``.
Changes
* CI transitioned to Azure Pipelines (from AppVeyor and Travis CI).
* Switched to ``pytest`` for running tests (from ``nose``).
* Bundled zstandard library upgraded from 1.3.8 to 1.4.3.
Version 3.1:
This will be last version with support for Python 2.x
New feature:
Accept pathlib objects as filenames.
Accept bytes filenames in Python 3
Fixes:
Use bug-compatible SHA1 for longer passwords (> 28 chars) in RAR3 encrypted headers.
Return true/false from _check_unrar_tool
Include all test files in archive
Include volume number in NeedFirstVolume exception if available (rar5).
Cleanups:
Convert tests to pytest.
v2.2.1:
Update the bundled LZ4 library to version 1.9.1
This release updates the bundled LZ4 library to version 1.9.1.
The 2.2.x releases will be the final release that support Python 2.7. In the near future we'll begin work on the 3.0.x release which will only support Python >= 3.5, and will require LZ4 > 1.9.0.
v2.2.0:
Add more detail to the install section of docs
v0.6.0
When adding implicit dirs, ensure that ancestral directories
are added and that duplicates are excluded.
The library now relies on more_itertools
5.61.0
KTar::openArchive: Don't assert if file has two root dirs
KZip::openArchive: Don't assert when opening broken files
5.60.0
Do not crash if the inner file wants to be bigger than QByteArray max size
5.59.0
Test reading and seeking in KCompressionDevice
KCompressionDevice: Remove bIgnoreData
KAr: fix out-of-bounds read (on invalid input) by porting to QByteArray
KAr: fix parsing of long filenames with Qt-5.10
KAr: the permissions are in octal, not decimal
KAr::openArchive: Also check ar_longnamesIndex is not < 0
KAr::openArchive: Fix invalid memory access on broken files
KAr::openArchive: Protect against Heap-buffer-overflow in broken files
KTar::KTarPrivate::readLonglink: Fix crash in malformed files
5.58.0
KTar: Protect against negative longlink sizes
Fix invalid memory write on malformed tar files
Fix memory leak when reading some tar files
Fix uninitialized memory use when reading malformed tar files
Fix stack-buffer-overflow read on malformed files
Fix null-dereference on malformed tar files
Install krcc.h header
Fix double delete on broken files
Disallow copy of KArchiveDirectoryPrivate and KArchivePrivate
Fix KArchive::findOrCreate running out of stack on VERY LONG paths
Introduce and use KArchiveDirectory::addEntryV2
removeEntry can fail so it's good to know if it did
KZip: fix Heap-use-after-free in broken files
LZ4 v1.9.2
fix : out-of-bound read in exceptional circumstances when using decompress_partial()
fix : slim opportunity for out-of-bound write with compress_fast() with a large enough input and when providing an output smaller than recommended (< LZ4_compressBound(inputSize))
fix : rare data corruption bug with LZ4_compress_destSize()
fix : data corruption bug when Streaming with an Attached Dict in HC Mode
perf: enable LZ4_FAST_DEC_LOOP on aarch64/GCC by default
perf: improved lz4frame streaming API speed
perf: speed up lz4hc on slow patterns when using external dictionary
api: better in-place decompression and compression support
cli : --list supports multi-frames files
cli: --version outputs to stdout
cli : add option --best as an alias of -12
misc: Integration into oss-fuzz
Zstandard v1.4.3
Dictionary Compression Regression
We discovered an issue in the v1.4.2 release, which can degrade the effectiveness of dictionary compression. This release fixes that issue.
Detailed Changes
* bug: Fix Dictionary Compression Ratio Regression
* bug: Fix Buffer Overflow in v0.3 Decompression
* build: Add support for IAR C/C++ Compiler for Arm
* misc: Add NULL pointer check in util.c by
The canonical form [1] of an R package Makefile includes the
following:
- The first stanza includes R_PKGNAME, R_PKGVER, PKGREVISION (as
needed), and CATEGORIES.
- HOMEPAGE is not present but defined in math/R/Makefile.extension to
refer to the CRAN web page describing the package. Other relevant
web pages are often linked from there via the URL field.
This updates all current R packages to this form, which will make
regular updates _much_ easier, especially using pkgtools/R2pkg.
[1] http://mail-index.netbsd.org/tech-pkg/2019/08/02/msg021711.html
2019-02-18 Stuart Caie <kyzer@cabextract.org.uk>
* chmd_read_headers(): a CHM file name beginning "::" but shorter
than 33 bytes will lead to reading past the freshly-allocated name
buffer - checks for specific control filenames didn't take length
into account. Thanks to ADLab of Venustech for the report and
proof of concept.
2019-02-18 Stuart Caie <kyzer@cabextract.org.uk>
* chmd_read_headers(): CHM files can declare their chunks are any
size up to 4GB, and libmspack will attempt to allocate that to
read the file.
This is not a security issue; libmspack doesn't promise how much
memory it'll use to unpack files. You can set your own limits by
returning NULL in a custom mspack_system.alloc() implementation.
However, it would be good to validate chunk size further. With no
offical specification, only empirical data is available. All files
created by hhc.exe have a chunk size of 4096 bytes, and this is
matched by all the files I've found in the wild, except for one
which has a chunk size of 8192 bytes, which was created by someone
developing a CHM file creator 15 years ago, and they appear to
have abandoned it, so it seems 4096 is a de-facto standard.
I've changed the "chunk size is not a power of two" warning to
"chunk size is not 4096", and now only allow chunk sizes between
22 and 8192 bytes. If you have CHM files with a larger chunk size,
please send them to me and I'll increase this upper limit.
Thanks to ADLab of Venustech for the report.
2019-02-18 Stuart Caie <kyzer@cabextract.org.uk>
* oabd.c: replaced one-shot copying of uncompressed blocks (which
requires allocating a buffer of the size declared in the header,
which can be 4GB) with a fixed-size buffer. The buffer size is
user-controllable with the new msoab_decompressor::set_param()
method (check you have version 2 of the OAB decompressor), and
also controls the input buffer used for OAB's LZX decompression.
Reminder: compression formats can dictate how much memory is
needed to decompress them. If memory usage is a security concern
to you, write a custom mspack_system.alloc() that returns NULL
if "too much" memory is requested. Do not rely on libmspack adding
special heuristics to know not to request "too much".
Thanks to ADLab of Venustech for the report.
Zstandard v1.4.2
Legacy Decompression Fix
This release is a small one, that corrects an issue discovered in the previous release. Zstandard v1.4.1 included a bug in decompressing v0.5 legacy frames, which is fixed in v1.4.2.
Detailed Changes
bug: Fix bug in zstd-0.5 decoder
bug: Fix seekable decompression in-memory API
bug: Close minor memory leak in CLI
misc: Validate blocks are smaller than size limit
misc: Restructure source files
1.0.8 (13 Jul 19)
~~~~~~~~~~~~~~~~~
* Accept as many selectors as the file format allows.
This relaxes the fix for CVE-2019-12900 from 1.0.7
so that bzip2 allows decompression of bz2 files that
use (too) many selectors again.
* Fix handling of large (> 4GB) files on Windows.
* Cleanup of bzdiff and bzgrep scripts so they don't use
any bash extensions and handle multiple archives correctly.
* There is now a bz2-files testsuite at
https://sourceware.org/git/bzip2-tests.git
1.0.7 (27 Jun 19)
~~~~~~~~~~~~~~~~~
* Fix undefined behavior in the macros SET_BH, CLEAR_BH, & ISSET_BH
* bzip2: Fix return value when combining --test,-t and -q.
* bzip2recover: Fix buffer overflow for large argv[0]
* bzip2recover: Fix use after free issue with outFile (CVE-2016-3189)
* Make sure nSelectors is not out of range (CVE-2019-12900)
v1.4.1
bug: Fix data corruption in niche use cases by @terrelln (#1659)
bug: Fuzz legacy modes, fix uncovered bugs by @terrelln (#1593, #1594, #1595)
bug: Fix out of bounds read by @terrelln (#1590)
perf: Improve decode speed by ~7% @mgrice (#1668)
perf: Slightly improved compression ratio of level 3 and 4 (ZSTD_dfast) by @cyan4973 (#1681)
perf: Slightly faster compression speed when re-using a context by @cyan4973 (#1658)
perf: Improve compression ratio for small windowLog by @cyan4973 (#1624)
perf: Faster compression speed in high compression mode for repetitive data by @terrelln (#1635)
api: Add parameter to generate smaller dictionaries by @tyler-tran (#1656)
cli: Recognize symlinks when built in C99 mode by @felixhandte (#1640)
cli: Expose cpu load indicator for each file on -vv mode by @ephiepark (#1631)
cli: Restrict read permissions on destination files by @chungy (#1644)
cli: zstdgrep: handle -f flag by @felixhandte (#1618)
cli: zstdcat: follow symlinks by @vejnar (#1604)
doc: Remove extra size limit on compressed blocks by @felixhandte (#1689)
doc: Fix typo by @yk-tanigawa (#1633)
doc: Improve documentation on streaming buffer sizes by @cyan4973 (#1629)
build: CMake: support building with LZ4 @leeyoung624 (#1626)
build: CMake: install zstdless and zstdgrep by @leeyoung624 (#1647)
build: CMake: respect existing uninstall target by @j301scott (#1619)
build: Make: skip multithread tests when built without support by @michaelforney (#1620)
build: Make: Fix examples/ test target by @sjnam (#1603)
build: Meson: rename options out of deprecated namespace by @lzutao (#1665)
build: Meson: fix build by @lzutao (#1602)
build: Visual Studio: don't export symbols in static lib by @scharan (#1650)
build: Visual Studio: fix linking by @absotively (#1639)
build: Fix MinGW-W64 build by @myzhang1029 (#1600)
misc: Expand decodecorpus coverage by @ephiepark (#1664)
Update ruby-zip to 1.2.3, here is release note.
1.2.3 (2019-05-23)
* Allow tilde in zip entry names #391 (fixes regression in 1.2.2 from #376)
* Support frozen string literals in more files #390
* Require pathname explicitly #388 (fixes regression in 1.2.2 from #376)
Tooling / Documentation:
* CI updates #392, #394
- Bump supported ruby versions and add 2.6
- JRuby failures are no longer ignored (reverts #375 / part of #371)
* Add changelog entry that was missing for last release #387
* Comment cleanup #385
Since the GitHub release information for 1.2.2 is missing, I will also include
it here:
1.2.2 (2018-09-01)
NB: This release drops support for extracting symlinks, because there was no
clear way to support this securely. See #376 (comment) for details.
* Fix CVE-2018-1000544 #376 / #371
* Fix NoMethodError: undefined method `glob' #363
* Fix handling of stored files (i.e. files not using compression) with general
purpose bit 3 set #358
* Fix close on StringIO-backed zip file #353
* Add Zip.force_entry_names_encoding option #340
* Update rubocop, apply auto-fixes, and fix regressions caused by said
auto-fixes #332, #355
* Save temporary files to temporary directory (rather than current directory)
#325
Tooling / Documentation:
* Turn off all terminal output in all tests #361
* Several CI updates #346, #347, #350, #352
* Several README improvements #345, #326, #321