Changes in 3.96 (23 Jan 2020):
* bug fixes - see https://github.com/upx/upx/milestone/5
Changes in 3.95 (26 Aug 2018):
* Flag --android-shlib to work around bad design in Android
* Flag --force-pie when ET_DYN main program is not marked as DF_1_PIE
* Better compatibility with varying layout of address space on Linux
* Support for 4 PT_LOAD layout in ELF generated by binutils-2.31
* bug fixes, particularly better diagnosis of malformed input
* bug fixes - see https://github.com/upx/upx/milestone/4
Changes in 3.94 (12 May 2017):
* Add support for arm64-linux (aka "aarch64").
* Add support for --lzma compression on 64-bit PowerPC (Thierry Fauck).
* For Mach, "upx -d" will unpack a prefix of the file (and warn).
* Various improvements to the ELF formats.
* bug fixes - see https://github.com/upx/upx/milestone/3
Changes in 3.93 (29 Jan 2017):
* Fixed some win32/pe and win64/pe regressions introduced in 3.92
* bug fixes - see https://github.com/upx/upx/milestone/2
Changes in 3.92 (11 Dec 2016):
* INFO: UPX has moved to GitHub - the new home page is https://upx.github.io
* IMPORTANT: all PE formats: internal changes: reunited the diverged source
files - please report all regressions into the bug tracker and try UPX 3.91
in case of problems.
* Support Apple MacOS 10.12 "Sierra", including more-robust de-compression.
* Explicitly diagnose Go-language bad PT_LOAD; recommend hemfix.c.
https://sourceforge.net/p/upx/bugs/195/https://github.com/pwaller/goupx
* Fix CERT-FI Case 829767 UPX command line tools segfaults.
Received by UPX Team on 2015-May-08; originally reported
by Codenomicon to NCSC-FI on 2015-01-08.
The vulnerabilities were discovered by Joonas Kuorilehto and
Antti Häyrynen from Codenomicon.
* bug fixes - see https://github.com/upx/upx/milestone/1
Update ruby-zip to 1.3.0, latest 1.x release.
1.3.0 (2019-09-25)
Security
* Add validate_entry_sizes option so that callers can trust an entry's
reported size when using extract #403
o This option defaults to false for backward compatibility in this
release, but you are strongly encouraged to set it to true. It
will default to true in rubyzip 2.0.
New Feature
* Add add_stored method to simplify adding entries without compression #366
Tooling / Documentation
* Add more gem metadata links #402
1.2.4 (2019-09-06)
* Do not rewrite zip files opened with open_buffer that have not changed #360
Tooling / Documentation
* Update example_recursive.rb in README #397
* Hold CI at trusty for now, automatically pick the latest ruby patch
version, use rbx-4 and hold jruby at 9.1 #399
New features with AN-2020-02-11:
- star: fixed typos in star.1 and star.4
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
- star: fixed typos in header.c
Thanks to Eric Ackermann for reporting
- star: translated some German comment to English in diff.c, extract.c,
list.c, remove.c, restore.c, fifo.c
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
- star: create.c: #ifdef __what_people_would_expect__ changed to:
#ifdef __wrong_but_what_people_would_expect__ for better readability.
- libschily: new man pages starthandlecond.3 and unhandlecond.3
- libschily: handlecond.3 and raisecond.3 now correctly mention
handlecond() & raisecond() instead of handle()/raise(). The old names
from 1980 had to be renamed because os an unfriendly actt from the
C standard commitee.
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
- libschily: various small fixes in various man pages from libschily.
Thanks to Eric Ackermann for reporting
as a side effect of a code review for SchilliX-ON
Release notes for SZIP 2.1.1
January 13, 2017
Bug fixes:
-- None.
New features:
-- Add CMake support.
Know problems:
-- On IRIX64-6.5, shared library version 3.0 is created instead of 2.0.
-- There is no support for shared library on the AIX 5.* systems.
User support:
Report all problems to help@hdfgroup.org.
For more information on SZIP, see:
http://www.hdfgroup.org/doc_resource/SZIP/
Changes in version 1.8:
A buffer overflow has been fixed in zcat which happened sometimes when
the '-v, --show-nonprinting' option was used (or indirectly enabled).
A canary byte has been added to the output buffer to prevent the buffer
overflow from happening again.
The option '-R, --dereference-recursive', which recursively follows
symbolic links, has been added to zcat, zgrep, ztest and zupdate.
The option '-r, --recursive' now skips symlinks that are encountered
recursively.
If no files are given to zcat, zgrep, ztest and zupdate, a recursive
search will now examine the current working directory.
Recursive directory loops are now detected.
zcat and zgrep now ignore directories given in the command line if
'--recursive' is not specified, instead of reporting an error.
Extra trailing slashes are now removed from directories given in the
command line before recursing into them.
zcat and zgrep now show the right error when they can't open an input
file instead of showing "No such file or directory".
Killed decompressors are now waited for, preventing failure caused by
too many open pipes.
Test and document that if a file fails to decompress, zcat, zgrep and
ztest continue processing the rest of the files.
Test and document that if an error happens while recompressing a file,
zupdate exits immediately without recompressing the rest of the files.
The configure script now accepts appending options to CXXFLAGS using the
syntax 'CXXFLAGS+=OPTIONS'.
This software has a lot of hard-coded references to external binaries,
with sometimes incorrect assumptions. Deal with these through the SUBST
framework for now. (Any mistakes here are solely mine.)
Arqiver is a simple Qt5 archive manager as a front-end for libarchive
(bsdtar), gzip and 7z.
Packaged in pkgsrc-wip by pin, as part of a broader effort toward
providing all of LXQt and associated utility applications. General
reviews, feedback, and testing of such by gdt@, ng0@, and myself.
Changes (from https://github.com/Parchive/par2cmdline/releases/tag/v0.8.1):
BlackEagle (3):
Add .DELETE_ON_ERROR to avoid corrupt files in workdir
split release scripts a bit because win stuff is not build on the same machine as *nix stuff anymore
bump 0.8.1
Ike Devolder (3):
Merge pull request #131 from mdnahas/master
Merge pull request #132 from mdnahas/library_dev
Merge pull request #134 from mdnahas/long_read_write
Michael Nahas (70):
BUG FIX: ^= operator was returning wrong value for powers of 1
.gitignore now ignores emacs backup files
FIX: typos, comments, indentation
Added unit tests for 5 classes
Changed all output to use parameters, rather than global variables.
Set chunksize to zero, in unspecified case
all commandline errors are first in par2creator::Process
moved reference to commandline to top of parXrepairer
removed largest file and total file size from commandline class
Removed ExtraFile::FileSize method --- this slows things slightly.
Convered ExtraFile to just a string.
grouped commandline options by operation
Added my name to list of authors
added the most common usage as an example
move num-of-threads setting code out of commandline
added links to websites explaining CRC operations
Moved NoiseLevel out of CommandLine
remove duplicate code in par2cmdline.cpp
Moved par1repairer into a function call
Moved par2repairer into a function call
Moved par2creator into a function call
Moved filethreads out of commandline.h/cpp
removed ExtraFile from commandline.h/cpp
removed ExtraFileIterator from commandline.h/cpp
Split up CommandLine::Parse soo I can move stuff in
Reordered checks in commandline.cpp to match order of variable declarations
Moved computation of blocksize into commandline.cpp
BUG FIX: when changed nthreads, forgot to add OPENMP #ifdef around it
Moved redundnacy calculation out of par2creator into commandline
FEATURE: On Linux, adjusts buffer sizes to half the memory of the system
commandline now treats argv aas const char * const *, for easier testing.
unit tests for CommandLine.Parse(), plus bug fixes to it.
Changed code to support a par2lib static library
DANGEROUS: Moved ifdefs from par2cmdline.h into par2lib.h
FEATURE: Created libpar2, a static library for par2.
Added unit tests for diskfile
BUG FIX: DiskFile::Delete() now sets 'exists' to false
BUG FIX: getting physical memory now works for OSX
DANGEROUS: Changed WIN32 to _WIN32 or _MSC_VER
merged code for ComputeRecoveryFileCount into libpar2
Added output to tests, to see they had done something.
BUG FIX - SQUASH - did not ifdef OPEN_MP around thread code
TEST FIX: newline were expanding into two chars on Windows
Added test for DiskFile::Create when file already exists
BUG FIX: DiskFile::Create now returns error when file already exists
BUG FIX: Should now implement Par2 standard for filenames
Test scripts now use .exe and wine, if exe exists
TEST FIX: Some tests did not return 1 on error.
DANGEROUS, BUG FIX: Fixed most things found by cppcheck
BUG FIX: fixed memory leak by FindFiles using std::unique_ptr
FEATURE: Modified test scripts to support valgrind
Changes made so that it compiles on OSX High Sierra
Uncommented line for AM_PROG_AR in configure.ac; needed for Windows
g++ is now forced to use C++ version 2011
Added ar-lib to .gitignore
Unit tests are now run in WINE and with Valgrind
BUILD FIX: Added tests/unit_tests to Makefile.am dist
Added sorting to CriticalPackets, for consistent output
Updated copyright information with my name
Added notes on Cppcheck, Valgrind, MinGW, and WINE to ROADMAP
Added information on WINEPATH variable to ROADMAP file
Made all unit tests look similar and say FAILED/SUCCESS
fixed typo in test
BUG FIX: Version was compared against opNone.
New Visual C++ project file, thanks to Anime Tosho
BUG FIX: had introduced bug handling Unicode. Reverted change.
Changed types to size_t to satisfy Microsoft compiler
Support file I/O on blocksize greater than 4GB
Merge pull request #135 from rwtodd/master
Merge pull request #136 from rwtodd/master
Richard Todd (4):
Used _stati64() for WIN32 DiskFile::FileExists()
Added par2cmdline.h to par2cmdline.cpp for DEBUG_NEW
fixed 'need signed i for openmp' errors on Windows
Renamed par2cmdline.h to libpar2internal.h.
build with qt5 5.14
All frameworks
Port from QRegExp to QRegularExpression
Port from qrand to QRandomGenerator
Fix compilation with Qt 5.15 (e.g. endl is now Qt::endl,
QHash insertMulti now requires using QMultiHash...)
Attica
Don't use a verified nullptr as a data source
Support multiple children elements in comment elements
Set a proper agent string for Attica requests
Baloo
Correctly report if baloo_file is unavailable
Check cursor_open return value
Initialise QML monitor values
Move URL parsing methods from kioslave to query object
Breeze Icons
Change XHTML icon to be a purple HTML icon
Merge headphones and zigzag in the center
Add application/x-audacity-project icon
Add 32px preferences-system
Add application/vnd.apple.pkpass icon
icon for ktimetracker using the PNG in the app repo, to be replaced
with real breeze SVG
add kipi icon, needs redone as a breeze theme svg [or just kill off kipi]
Extra CMake Modules
[android] Fix apk install target
Support PyQt5 compiled with SIP 5
Framework Integration
Remove ColorSchemeFilter from KStyle
KDE Doxygen Tools
Display fully qualified class/namespace name as page header
KCalendarCore
Improve README.md to have an Introduction section
Make incidence geographic coordinate also accessible as a property
Fix RRULE generation for timezones
KCMUtils
Deprecate KCModuleContainer
KCodecs
Fix invalid cast to enum by changing the type to int rather than enum
KCompletion
Deprecate KPixmapProvider
[KHistoryComboBox] Add method to set an icon provider
KConfig
kconfig EBN transport protocol cleanup
Expose getter to KConfigWatcher's config
Fix writeFlags with KConfigCompilerSignallingItem
Add a comment pointing to the history of Cut and Delete sharing a shortcut
KConfigWidgets
Rename "Configure Shortcuts" to "Configure Keyboard Shortcuts"
KContacts
Align ECM and Qt setup with Frameworks conventions
Specify ECM dependency version as in any other framework
KCoreAddons
Add KPluginMetaData::supportsMimeType
[KAutoSaveFile] Use QUrl::path() instead of toLocalFile()
Unbreak build w/ PROCSTAT: add missing impl. of KProcessList::processInfo
[KProcessList] Optimize KProcessList::processInfo
[KAutoSaveFile] Improve the comment in tempFileName()
Fix KAutoSaveFile broken on long path
KDeclarative
[KeySequenceHelper] Grab actual window when embedded
Add optional subtitle to grid delegate
[QImageItem/QPixmapItem] Don't lose precision during calculation
KFileMetaData
Partial fix for accentuated characters in file name on Windows
Remove unrequired private declarations for taglibextractor
Partial solution to accept accentuated characters on windows
xattr: fix crash on dangling symlinks
KIconThemes
Set breeze as default theme when reading from configuration file
Deprecate the top-level IconSize() function
Fix centering scaled icons on high dpi pixmaps
KImageFormats
pic: Fix Invalid-enum-value undefined behaviour
KIO
[KFilePlacesModel] Fix supported scheme check for devices
Embed protocol data also for Windows version of trash ioslave
Adding support for mounting KIOFuse URLs for applications that don't use KIO
Add truncation support to FileJob
Deprecate KUrlPixmapProvider
Deprecate KFileWidget::toolBar
[KUrlNavigator] Add RPM support to krarc:
KFilePlaceEditDialog: fix crash when editing the Trash place
Add button to open the folder in filelight to view more details
Show more details in warning dialog shown before starting a
privileged operation
KDirOperator: Use a fixed line height for scroll speed
Additional fields such as deletion time and original path are now
shown in the file properties dialog
KFilePlacesModel: properly parent tagsLister to avoid memleak.
HTTP ioslave: call correct base class in virtual_hook(). The
base of HTTP ioslave is TCPSlaveBase, not SlaveBase
Ftp ioslave: fix 4 character time interpreted as year
Re-add KDirOperator::keyPressEvent to preserve BC
Use QStyle for determining icon sizes
Kirigami
ActionToolBar: Only show the overflow button if there are visible
items in the menu
Don't build and install app templates on android
Don't hardcode the margin of the CardsListView
Add support for custom display components to Action
Let the other components grow if there's more things on the header
Remove dynamic item creation in DefaultListItemBackground
reintroduce the collapse button
Show application window icon on AboutPage
KItemModels
Add KColumnHeadersModel
KJS
Added tests for Math.exp()
Added tests for various assignment operators
Test special cases of multiplicate operators (*, / and %)
KNewStuff
Ensure the dialog title is correct with an uninitialised engine
Don't show the info icon on the big preview delegate
Support archive installs with adoption commands
Send along the config name with requests
KPeople
Expose enum to the metaobject compiler
KQuickCharts
Also correct the shader header files
Correct license headers for shaders
KService
Deprecate KServiceTypeProfile
KTextEditor
Add "line-count" property to the ConfigInterface
Avoid unwanted horizontal scrolling
KWayland
[plasmashell] Update docs for panelTakesFocus to make it generic
[plasmashell] Add signal for panelTakesFocus changing
KXMLGUI
KActionCollection: provide a changed() signal as a replacement for removed()
Adjust keyboard shortcut configuration window's title
NetworkManagerQt
Manager: add support for AddAndActivateConnection2
cmake: Consider NM headers as system includes
Sync Utils::securityIsValid with NetworkManager
Plasma Framework
[ToolTip] Round position
Enable wheel events on Slider {}
Sync QWindow flag WindowDoesNotAcceptFocus to wayland plasmashell interface
[calendar] Check out of bounds array access in QLocale lookup
[Plasma Dialog] Use QXcbWindowFunctions for setting window types Qt
WindowFlags doesn't know
[PC3] Complete plasma progress bar animation
[PC3] Only show progress bar indicator when the ends won't overlap
[RFC] Fix Display Configuration icon margins
[ColorScope] Work with plain QObjects again
[Breeze Desktop Theme] Add monochrome user-desktop icon
Remove default width from PlasmaComponents3.Button
[PC3 ToolButton] Have the label take into account complementary color schemes
Added background colors to active and inactive icon view
QQC2StyleBridge
[ToolTip] Round position
Update size hint when font changes
Solid
Display first / in mounted storage access description
Ensure mounted nfs filesystems matches their fstab declared counterpart
Sonnet
The signal done is deprecated in favour of spellCheckDone, now correctly emitted
Syntax Highlighting
LaTeX: fix brackets in some commands
TypeScript: add "bigint" primitive type
Python: improve numbers, add octals, binaries and "breakpoint" keyword
SELinux: add "glblub" keyword and update permissions list
Several enhancements to gitolite syntax definition
v1.1.0
======
#32: For read-only zip files, complexity of ``.exists`` and
``joinpath`` is now constant time instead of ``O(n)``, preventing
quadratic time in common use-cases and rendering large
zip files unusable for Path. Big thanks to Benjy Weinberger
for the bug report and contributed fix (#33).
pkglint -r --network --only "migrate"
As a side-effect of migrating the homepages, pkglint also fixed a few
indentations in unrelated lines. These and the new homepages have been
checked manually.
1.6.0 [2020-01-24]
==================
* Avoid using umask() since it's not thread-safe.
* Set close-on-exec flag when opening files.
* Do not accept empty files as valid zip archives any longer.
* Add support for XZ compressed files (using liblzma).
* Add support for cancelling while closing zip archives.
* Add support for setting the time in the on-disk format.
This library is for working with ".tar" archive files. It can read and
write a range of common variations of archive format including V7,
POSIX USTAR and GNU formats.
It provides support for packing and unpacking portable archives. This
makes it suitable for distribution but not backup because details like
file ownership and exact permissions are not preserved.
It also provides features for random access to archive content using
an index.
pkglint --only "https instead of http" -r -F
With manual adjustments afterwards since pkglint 19.4.4 fixed a few
indentations in unrelated lines.
This mainly affects projects hosted at SourceForce, as well as
freedesktop.org, CTAN and GNU.
Update ruby-mixlib-archive to 1.0.5.
1.0.5 (2019-12-30)
* Remove Travis and update Github templates #40 (tas50)
* Chefstyle fixes + test on Windows in Buildkite #43 (tas50)
* Test on Ruby 2.7rc and other testing updates #47 (tas50)
* Substitute require for require_relative #48 (tas50)
Changes from 1.0.2 to 1.0.4 are unknown.
Update ruby-ffi-libarchive to 1.0.0.
* Update HOMEPAGE and LICENSE.
* Proper COMMENT.
* No need to use gmake.
* add "USE_LANGUAGES= # none".
1.0.0 (2019-12-28)
* Chefstyle fixes to get the build green again #25 (tas50)
* Add support for custom read functions #27 (jatoben)
* Add Ruby 2.7rc testing and cache gem installs in Buildkite #28 (tas50)
The zip-archive library provides functions for creating, modifying,
and extracting files from zip archives. The zip archive format is
documented in http://www.pkware.com/documents/casestudies/APPNOTE.TXT.
Certain simplifying assumptions are made about the zip archives: in
particular, there is no support for strong encryption, zip files that
span multiple disks, ZIP64, OS-specific file attributes, or
compression methods other than Deflate. However, the library should be
able to read the most common zip archives, and the archives it
produces should be readable by all standard unzip programs.
Change log:
* Translations update
* fr-command-rpm: Use rpm2cpio binary shipped with rpm package
* add source rpm to supported mime-types
* use cpio if no binary is found
* add a configure environment variable to select the cpio path
* help: pull updated Russian translation to remove broken one
* Fix localized pause UI logic
Based on Schily Tools Release 2019-12-05.
AN-2019-11-11:
- libschily: the new functions wcastoi wcastol wcastolb that have been
added with the previous release have been forgotten in the linker map
file.
- star: When using star -copy, the default is now -no-secure-links.
This is what users like since they are copying their own data that
per definition cannot be untrusted external data.
- star: star -diff is now "more smooth" to use, when the archive type
is GNU TAR and the "atime" or "ctime" properties are not present
for a specific file. Before, Jan 1 1970 was the assumed time in
the archive.
- star: star -diff now prints a diff statistics summary.
- star: New version date
AN-2019-12-05:
- include/schily/stdio.h: We no longer define getc_unlocked() on Solaris
in case that it is defined already. This was needed since Oracle
Solaris 11.4 compiles 64 bit by default and made stdio less opaque
than before - this result in the same bahavior as for 32 bit programs.
This means, we no longer need to use tricks to get fast access to
stdio.
Thanks to Rolf Dietze for reporting.
- libschily: the function getargs() and similar (from the option parser
group) fixed a bug that prevented the usability of the official method
to include the char '+' inside option names by quoting the character
in the option name used in the format string.
This bug has been fixed in order to be able to support "star -dump+ ..."
- libshedit: because of the changes in Oracle Solaris 11.4, we need to
#undef FAST_GETC_PUTC in order to avoid to use getc_unlocked().
Thanks to Rolf Dietze for reporting.
- libfind: A new function walksname() has been added to set up the
internals in "struct WALK" for external path names. This feature is
needed to support "star -c list=names -find ...".
- libfind: The functions fetchdir()/dfetchdir(), sortdir(), cmpdir() now
use size_t instead of int for the entry number count and the allocated
size.
This breaks binary compatibility in 64 bit mode for those users only,
that use the interfaces fetchdir()/dfetchdir(), sortdir(), cmpdir().
The only application that currently uses these interfaces is "star".
Simple users of libfind and programs compiled as 32 bit programs
are not affected by the change.
Warning: packetizers need to take care of the changed name of the
shared library file for libfind.
- libfind: Because of the binary incompatible change, we incremented
the major library version number from 3 to 4.
WARNING: This affects only distros that ship 64 bit versions of star
using a shared version of libfind.
Since Solaris defaults to 32 bit programs, this is expected to only
affect Linux distros.
- libstreamar: unicode.c has been changed to make sure that is works even
in case that the "len" parameter is an unsigned.
- libstreamar: to_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- libstreamar: from_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: The new-volume-script= script is now called at the end of
_every tape_, even at the last one. This now allows to have a script
that automatically renames a standard archive name into numbered
archive names when a tree should be splitted into chunks, e.g. with
this command line:
star -c f=/tmp/v.tar tsize=1G new-volume-script='sh -c \
"mv /tmp/v.tar /tmp/v$(($1-1)).tar" nv' .
you get partial archives named /tmp/v1.tar ... /tmp/vn.tar
Note that this would not work in case that "sh" is the historical
Bourne Shell, e.g. as seen on Solaris 10. This is because
"shell arithmetics" is a feature that was not present in the historical
Bourne Shell.
- star: The man page now mentions that "star cli=tar ..." works as an
alias to "star cli=suntar ...".
- star: The man page now mentions that "star cli=gnutar ..." does not
impement the bugs from the GNU option parser.
- star: Is "star cli=xxx ..." is used, then star now calls
set_progname(xxx) and thus prints error messages labelled with the
command line interface name.
- star: In "pax" command line mode, star now uses -no-fsync as default
in order to behave the same way as the closed source Solaris pax
does.
- star: Star now allows to combine the option list=names with -find
This is useful, whenever a list of filenames is longer than ARG_MAX and
thus cannot be used in the star command line after -find.
Here is an example on how to use the new feature in a mercurial.
based development environment:
(hg status -a; hg status -m; hg status -c) | sed -e 's/^. //' > .hgfiles
star -c list=.hgfiles -find -mtime -100 > /tmp/new.tar
The first command puts all files under control of mercurial into
thw file ".hgfiles" , while omitting the compile results from the
file ".hgfiles".
The second command takes that list and puts only those files into
the archive that have been modified during the past 100 days.
- star: A new option "-dump+" allows to have the "SCHILY.volhdr.hostname"
meta data field filled out even when not making dumps with an
associated dump level. This helps to remember the origin of tar
archives.
- star: unicode.c has been changed to use the same comment as unicode.c
in libstreamar.
- star: unicode.c has been changed to make sure that is works even.
in case that the "len" parameter is an unsigned.
- star: to_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: from_utf8() from unicode.c has been rewritten to use size_t.
instead of int. Several changes have been introduced in the other
code as well.
- star: due to the incompatible interface change in libfind (see
above), several changes have been introduced in order to use size_t
instead of int for fetchdir()/dfetchdir(), sortdir(), cmpdir().
This finally permits star to be able to make use of the current POSIX
maximum meta data size for so called "pax" archives (TAR archives
with POSIX.1-2001 meta data enhancements derived from the Solaris 7
concept for tar from 1997). Previous versions of star have been
limited to data no longer than 2 GB (even though the star main code
supports 8 GB file size), this version now supports 8 GB for the.
sum of all meta data for a single file in case a 64 bit binary from
star is used.
- star: the fifo now has a new variant of te function runfifo() that
is used by the new fifo(1) standalone program. See below.
- star: The README now mentions the new "star cli= ..." method and
the fact that star deals with infinite path name length.
0.6.2.1
* Add support for GHC 8.8 / base-4.13
0.6.2
* New cabal flag 'pkg-config' for discovering 'zlib` via pkg-config(1) (#16)
* Use CApiFFI where available for cross-compile friendliness (#14)
* Change the window bits range from 8..15 to 9..15 (#11)
0.6.1.2
* Fix a segfault when reading the stream multithreaded, #7
* New experimental cabal flag 'non-blocking-ffi' for 'safe' ffi calls
0.6.1.1
* Fixed building with GHC 7.0 and 7.2
0.6.0.2
* Fixed building with GHC 7.0 and 7.2
0.6.1.0
* Support for concatenated gzip files (multiple back-to-back streams)
0.6.0.1
* Fixed building with older GHC
* Fixed warnings with new GHC
* Fixed building on Windows
* Fixed testsuite
0.6.0.0
* New incremental interface for compression and decompression
* Provide access to unconsumed trailing data
* Simplified structured error type, and instance of Exception
* Updated bundled zlib C code to 1.2.8 (used on Windows)
* Fixed memory leak of zlib z_stream C structure
* More derivied instances (Eq, Show, Typeable, Generic)
0.5.4.2
* Builds with GHC 7.10
Changes 0.13.0:
* ``pytest-xdist`` ``pytest`` extension is now installed so tests can be
run in parallel.
* CI now builds ``manylinux2010`` and ``manylinux2014`` binary wheels
instead of a mix of ``manylinux2010`` and ``manylinux1``.
* Official support for Python 3.8 has been added.
* Bundled zstandard library upgraded from 1.4.3 to 1.4.4.
* Python code has been reformatted with black.
At least ZSTD_FRAMEHEADERSIZE_MIN and ZSTD_FRAMEHEADERSIZE_PREFIX
macros in 1.4.4 needs a `format' argument but py-zstandard is not
adapted to 1.4.4 yet.
PKGREVISION++
Brotli is a generic-purpose lossless compression algorithm that compresses data
using a combination of a modern variant of the LZ77 algorithm, Huffman coding
and 2nd order context modeling, with a compression ratio comparable to the best
currently available general-purpose compression methods. It is similar in speed
with deflate but offers more dense compression.
This package contains Python bindings for Brotli.
2.9:
Changes to the library:
optimized the ArchiveEntry class
added support for the zstd format
fixed the top-level __all__ list (7b97d8b)
updated the code to support Python 3.8
improved the add_file_from_memory method
lowered the level of log messages from warning to info
Tests:
dropped testing with Python 3.4, added 3.7 and 3.8
fixed unicode handling in entry tests
Upstream changes:
0.86 Thu Dec 5 20:53:07 GMT 2019
* Fix RT#131072 which was due to tar.exe and CRLF line endings
0.84 Wed Dec 4 21:20:04 GMT 2019
* Fix RT#119084 - won't compile on MSWin32 with taint
0.82 Fri Nov 22 19:09:35 GMT 2019
* Correctly compare version string for Archive::Tar (#7)
* Remove duplicate entry from the MANIFEST file. (#6)
* Treat minix the same as NetBSD
innoextract 1.8 (2019-09-15)
- Added support for Inno Setup 6.0.0 installers
- Added support for pre-release Inno Setup 5.6.2 installers used by GOG
- Added support for two modified Inno Setup 5.5.7 variants
- Added support for Inno Setup 1.3.0 to 1.3.23
- Added support for My Inno Setup Extensions installers older than 3.0.6.1
- Added support for modified Inno Setup variants using an alternative setup loader magic
- Added support for using boost_{zlib,bzip2} when statically linking Boost
- Added support for automatically reading external setup.0 files
- Encoding for non-Unicode installers is now determined from the languages supported by the installer, overridable using the --codepage option
- Implemented parsing of GOG Galaxy architecture constraints
- The architecture-specific suffixes @32bit and @64bit are now used to disambiguate colliding files
- Fixed extracting files from slices larger than 2 GiB with 32-bit builds
- Fixed output path for files with absolute paths (canonicalization now strips all unsafe characters)
- Fixed output directory being created even when not extracting files
- Fixed a hang when using the --language option
- Improved checksum verification for files reconstructed from GOG Galaxy file parts
- Changed header parsing to select the first version without warnings and failing that the first without errors
- Changed filesystem and output encoding to WTF-8 (extended UTF-8) to represent broken UTF-16 data
Based on schilytools release from 2019-10-25.
Changelog
=========
- libstreamer: Added new keywords "fsdevmajor" and "fsdevminor" as
there are too many incompatible st_dev binary formats.
These keywords are not yet created, but they are understood.
This permits to switch to the new format (fade out "dev") later.
- star: Added new keywords "SCHILY.fsdevmajor" and "SCHILY.fsdevminor"
as there are too many incompatible st_dev binary formats.
These keywords are not yet created, but they are understood.
This permits to switch to the new format (fade out "SCHILY.dev") later.
- star: The amount of shared mamory used by default has been reduced to
1 MB on Ultrix. These are old small machines, that do not support 8 MB.
Zstandard v1.4.4
This release includes some major performance improvements and new CLI features, which make it a recommended upgrade.
perf: Improved decompression speed, by > 10%
perf: Better compression speed when re-using a context
perf: Fix compression ratio when compressing large files with small dictionary
perf: zstd reference encoder can generate RLE blocks
perf: minor generic speed optimization
api: new ability to extract sequences from the parser for analysis
api: fixed decoding of magic-less frames
api: fixed ZSTD_initCStream_advanced() performance with fast modes, reported by @QrczakMK
cli: Named pipes support
cli: short tar's extension support
cli: command --output-dir-flat=DIE , generates target files into requested directory
cli: commands --stream-size=# and --size-hint=#
cli: command --exclude-compressed
cli: faster -t test mode
cli: improved some error messages
cli: fix rare deadlock condition within dictionary builder
build: single-file decoder with emscripten compilation script
build: fixed zlibWrapper compilation on Visual Studio, reported by @bluenlive
build: fixed deprecation warning for certain gcc version, reported by @jasonma163
build: fix compilation on old gcc versions
build: improved installation directories for cmake script, by Dmitri Shubin
pack: modified pkgconfig, for better integration into openwrt, requested by @neheb
misc: Improved documentation : ZSTD_CLEVEL, DYNAMIC_BMI2, ZSTD_CDict, function deprecation, zstd format
misc: fixed educational decoder : accept larger literals section, and removed UNALIGNED() macro
(from: HomePage)
0.37 (2019/08/24):
Fixed issue 14 - https://github.com/glasswalk3r/archive-tar-wrapper-perl/issues/14
0.36 (2019/04/11):
Bug fixes since tests on Windows were failing after latests changes.
Refactored code due unexpected way bsdtar on Windows and tar on OpenBSD behave.
Refactored the tests due the bugs found.
Small fixes on Pod, specially the section for MS Windows users.
Added the missing tests to the MANIFEST.
0.35 (2019/04/05):
Added _rem_dots method, a improved way to remove '.' and '..' from readdir(), supposed to be faster than previous implementation with sort() and shift().
Added two more tests to increase testing coverage.
Fixed a small bug regarding Pod, that now covers 100% of the code.
Small fixes to documentation.
0.34 (2019/03/22):
Increased test coverage.
Refactored code to make it easier for testing.
Disabled performance-small.t for now.
Refactored bzip2 tests.
Adding META.yml "provides" from Makefile.PL
Covered methods that didn't have Pod.
Refactored all Pod following best practices.
Refactored a single method to be "private".
Slightly changed new() to enable unit testing for OS specific configurations.
zipstream.py is a zip archive generator based on python 3.3's
zipfile.py. It was created to generate a zip file generator for
streaming (ie web apps). This is beneficial for when you want to
provide a downloadable archive of a large collection of regular files,
which would be infeasible to generate the archive prior to downloading
or of a very large file that you do not want to store entirely on disk
or on memory.
The archive is generated as an iterator of strings, which, when
joined, form the zip archive.
Based on Release 2019-10-07.
Changelog
=========
- configure: Some shells report a syntax error with "< file (cmd)"
and need the redirection statement to be *after* the command. Our
changes to support the V7 shell by adding round braces caused ash
variants like "dash" to fail.
Thanks to Harald van Dijk for reporting
- cont/cc-config.sh: canged some :>some-file statements into
(:)>some-file. they have meen missed when introducing work arounds
for the V7 Shell on Ultrix that does not support I/O redirection
for builtin commands.
Thanks to Robert Clausecker for reporting
- libschily/resolvepath.c: resolving a symlink that points to another
symlink that points to itself, caused a coredump as a result from an
endless recursion.
We now detect this situation and abort the check before the endless
recursion causes a stack overflow. A symlink that directly loops
is immediately stopped. A longer symlink loop chain over more than one
symlink can only be detected by the recursion nesting level and is
aborted after a nesting level of 1024 has been reached. This works
under the assumption that the minimum stack size is more than
1024 * PATH_MAX and that there is no useful directory path with more
than 1024 symlinks in the path.
----> This problem affected star and SCCS.
Thanks to Philipp Wellner for reporting
- star: Added a hint to the man page that helps to find pkglist= as a.
similar option to list=
- star: The new method to avoid extracting symlinks that point outside
the star working directory that has been introduced in October 2018
could cause a core dump if a symlink is checked that points to
another aready existing symlink that points to itself. This was caused
by a problem in libschily/resolvepath.c, see above.
Thanks to Philipp Wellner for reporting
- star: The option -no-secure-links now may be configured as a global
default via the tag STAR_SECURE_LINKS= in the file /etc/default/star
and as a private default via an environment of the same name.
If the value for this tag is 'n' or 'N', -no-secure-links is made the
default, any other value sets the option -secure-links as the default.
This may be useful for sysadmins that frequently use star to copy
installation specific files, but it is risky in case that alien TAR
archives are imported. The good news is that this permits users to
switch to the old star behavior where no checks for risky links
existed.
Thanks to Dennis Clarke for reporting
- star: A new enviroment STAR_NOHINT has been introduced to supress
hint messages that are otherwise seen in case STAR_SECURE_LINKS or
STAR_FSYNC is in the environment or in /etc/default/star
- star: New version date
Update archivers/ruby-minitar to 0.9.
## 0.9 / 2019-09-04
* jtappa added the ability to skip fsync with a new option to Minitar.unpack
and Minitar::Input#extract_entry. Provide `:fsync => false` as the last
parameter to enable. Merged from a modified version of PR [#37][].
## 0.8 / 2019-01-05
* inkstak resolved an issue introduced in the fix for [#31][] by allowing
spaces to be considered valid characters in strict octal handling. Octal
conversion ignores leading spaces. Merged from a slightly modified version
of PR [#35][].
* dearblue contributed PR [#32][] providing an explicit call to #bytesize for
strings that include multibyte characters. The PR has been modified to be
compatible with older versions of Ruby and extend tests.
* Akinori MUSHA (knu) contributed PR [#36][] that treats certain badly
encoded regular files (with names ending in `/`) as if they were
directories on decode.
## 0.7 / 2018-02-19
* Fixed issue [#28][] with a modified version of PR [#29][] covering the
security policy and position for Minitar. Thanks so much to ooooooo\_q for
the report and an initial patch. Additional information was added as
[#30][].
* dearblue contributed PR [#33][] providing a fix for Minitar::Reader when
the IO-like object does not have a `#pos` method.
* Kevin McDermott contributed PR [#34][] so that an InvalidTarStream is
raised if the tar header is not valid, preventing incorrect streaming of
files from a non-tarfile. This is a minor breaking change, so the version
has been bumped accordingly.
* Kazuyoshi Kato contributed PR [#26][] providing support for the GNU tar
long filename extension.
* Addressed a potential DOS with negative size fields in tar headers
([#31][]). This has been handled in two ways: the size field in a tar
header is interpreted as a strict octal value and the Minitar reader will
raise an InvalidTarStream if the size ends up being negative anyway.
ADVANCECOMP VERSION 2.1 2018/02
===============================
* Support ZIPs with data descriptor signature.
* Fixed a crash condition with invalid ZIP data.
ADVANCECOMP VERSION 2.0 2017/06
===============================
* Added support for reading MNG files with depth of 1, 2, and 4 bits.
* Added 64 bits binary for Windows.
* Updated to libdeflate 29-May-2017.
From https://github.com/ebiggers/libdeflate
at commit 1726e9e87fb6f98682dfdea2356d5ee58881fe7b.
Switch to latest distfile.
Update PLIST.
Replaced MESSAGE with share/doc/star/INSTALL.pkgsrc.
Changelog
=========
Release 2019-03-29:
- libschily: the lutimens() emulation no longer returns ENOSYS in case
that the lstat() call fails, since this error code may e.g. be
ENAMETOOLONG and used as an important indicator for long path name
handling.
This bug caused incorrect behavior when star extracted long pathnames
on a platform without utimensat().
- star: The code restructuring to openat() from Summer 2018 did not
only cause a noticeable speed up even when not using -find, it at
the same time resulted in a wrong error message when a file type
was seen that is not archivable (e.g. a socket while using the USTAR
format). The correct message was something like "Unsupported filetype",
while the bug caused a "file not found " message. This has been fixed
by adding a new parameter "fd" to the function stat_to_info().
- star: header.c::get_xhtype() we did forgot to initialize:
finfo.f_devminorbits
and
finfo.f_xflags
This is now done.
- star: The USTAR format now is able to create base-256 values in
the field "t_devmajor". "t_devminor" did already support base-256.
- star: The GNU tar format now is able to create base-256 values in
the fields "t_devmajor" and "t_devminor".
- star: The man page star.4 now mentions which fileds may have
base-256 numbers.
- star: The man page star.4 now longer contains the doubled "field field"
in the explanation of the added UID/GID number for ACLs.
- star: The man page star.4 now has a better description for the additional
numeric fields in the ACL entries to prevent GNU tar from continuing
with it's incompatible ACL implementation.
- star: New ACL reference archives for the ultra compact format, that has
been defined together with the libachive people, have been added to the
directory testscripts/:
acl-nfsv4-compact-test.tar.gz
acl-nfsv4-compact-test2.tar.gz
acl-nfsv4-compact-test3.tar.gz
acl-nfsv4-compact-test4.tar.gz
acl-nfsv4-compact-test5.tar.gz
- star: Linux now uses the new <linux/fs.h> instead of <ext2/ext2_fs.h>
to retrieve file flags.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for the following new BSD Flags:
compressed hidden offline rdonly reparse sparse system
has been added.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for the following new Linux flags:
dirsync nocow notail projinherit topdir
has been added.
Thanks to a hint from Martin Matuska <martin@matuska.org> from the
libarchive team.
- star: Support for reading non-comliant libarchive tarballs that use.
"securedeletion" or "journal"
instead of the documented text has been added. This was used by
libarchive until March 20, when libarchive has been fixed. We introduce
the above names for compatibility with old TARs created by libarchive.
- star: When an "old star" archive is read and this archive is from a system
like FreeBSD with non-continous minor bits, star no longer warns unless
there is a device file in the archive.
- star: unit tests: the scripts now contain
LC_ALL=C export LC_ALL
instead of just LC_ALL=C to make the environment exported.
- star/libstrar: The case where iconv() returns -1 and sets errno to E2BIG
is now handled correctly.
- star: The testscripts/ directory now contains the tar test archives
from the portability tests from Michal Gorny. Check
http://cdrtools.sourceforge.net/private/portability-of-tar-features.html
for an updated (to match star-1.6) variant of the results from the portability
tests from Michal Gorny.
- star: older versions of star did not print the messages:
"WARNING: Archive is 'xxx' compressed, trying to use the -xxx option.\n"
in case that the option -print-artype was in use. Later versions that
added support for more than "gzip" and "bzip2" forgot to add the same
exception for the new compression methods. We now ommit this message
for all compression types.
- star: Support for the "lzma" compression has been added.
- star: Support for the "freeze2" compression has been added.
- star: The compression method name list did not include the text "zstd".
We added the missing text.
- star: The hint messages that are printed when a compressed input archive
is not seekable did forgot to mention "lzip" and "zstd". We added the
forgotten messages.
- star: New unit tests check whether star is able to auto-detect and auto
decompress various compression formats.
- star: The unit tests now include the portability tests from Michal Gorny
- A new option cli=name (must be argv[1]) allows to select all
supported command line interfaces (star, suntar, gnutar,
pax, cpio) when called as star.
This is needed to be able to test all command line interfaces from
out unit tests since star is not installed in this case and a selection
from argv[0] would not work.
- star: The version date is now "2019/03/20"
Release 2019-04-29:
- libfind/star: the verbose listing code has ben restructured to have
the file permissions in the same string as the file type.
This is needed to implement the POSIX pax listformat interpreter
format %M in future.
- libfind/star: the verbose listing no longer prints "l" for mandatory
record locking for non-directory type file, but rather only
for plain files.
- star: The "gnutar" emulation now prints the --help output to stdout as
GNU tar does.
- star: The gnutar.1 man page now mentions that the GNU tar
options -g / -G did never work and it thus makes no sense to
implement them.
- star: very outdated code in list.c has been removed.
- star: Added new unit tests for incremental backups and restores.
This in special include tests that always fail with GNU tar
as GNU tar is not usable and never was usable for incremental
restores in case that the differences are more than trivial.
- star: new version date 2019-04-01
Release 2019-06-13:
- star: fixed a bug in the FIFO code that mainly happened on Linux (with
a 1000x higher probability than it happens on Solaris). The bug was.
caused, as a check for a flag has been done twice instead of only once
where it could change it's value between both locations. As a.
result, star reported "star: Implementation botch: with FIFO_MEOF"
as the tar side of the FIFO did sometimes not wait for the FIFO_IWAIT
state when called as "star -multivolume -tv f=... f=... ...
For this reason, star incorrectly got a wakeup at the wrong location.
The bug appeared in case that star -x/-t -multivol f=.. f=.. ....
has been called with very small tar archives.
Thanks to Heiko Eissfeldt for reporting.
- star: The FIFO code renamed the "flag" member of the "m_head" structure
to "gflag" for better readability.
- star: The debugging code in the FIFO has been enhanced to print the
names of the flag bits in addition to the hex values.
- star: A deadlock situation that happens once every 500000 tries on
Linux with multi-volume archives has been fixed.
The problem was caused by a complex condition where the get side
of the FIFO needs to check the EOF FIFO flag and the amount of data
available in the FIFO and then decide whether to wait for a wakeup or
not.
Since the EOF flag needs to be checked first, a context switch in
the get side of the FIFO could allow the put side to set the EOF flag
before the get side did check the fill ratio of the FIFO. This
caused both the get side and the put side to wait for a wakeup.
The new code introduced a new variable mp->mayoblock that is set by
the get side before checking for EOF. This new flags allows the
put side to know that the get side is just in a critical situation
and lets the put side wait until mp->mayoblock is no longer set,
which signals a stable state in the get side. This permits to
avoid the deadlock.
- star: Note that the FIFO has been initially written as a lock free
design in the late 1980's. This is to allow high portability to even
older UNIX versions. The star FIFO works on all UNIX variants that
support pipes and shared memory, which is e.g. the case for
SunOS-4.0 from 1988. At the time the FIFO has been designed, the
target OS did not support multi-CPU systems and problems in the
FIFO first appeared with massively faster multi-CPU systems around
y2000. The recently detected problems all have been triggered by a
different context switch behavior on Linux, even though they could
have appeared on any OS in case that many million tries are
attempted.
- star: bumped version to 1.6.1
Release 2019-07-15:
- star: the compress unit test no longer fails on Cygwin because the
gzip binary is compiled incorrectly and does not support LZW.
The related test is now skipped.
Thanks to Heiko Eissfeldt for reporting
- star: some new unit tests failed if the schilytools source tree has
been installed in a directory with spaces in it's name.
Thanks to Heiko Eissfeldt for reporting
- star: The unit tests for incremental backups include archives that
include userid/groupid and username/groupname that may not be
restorable on a different computer. We now ignore these ID meta.
data when comparing the results.
Thanks to Heiko Eissfeldt for reporting
- star: When comparing nanoseconds in time stamps, star now has a mode
that treats time stamps as equal in case that tv_nsec % 100 == 0
and the rest of the nano seconds is equal. This is needd on Cygwin
since NTFS counts in 1/10 microseconds since Januar 1 1601.
This applies to both star -diff and the "newer" check while extracting
files.
Thanks to Heiko Eissfeldt for reporting
- star: a new option diffopts=dnlink has been implemented to support
filesystems that do not follow the historic UNIX model for hard links
on directories.
Since Cygwin usually has a linkcount of 1 on directories, you need
to use "star -diff diffopts=!dnlink ..." if you like to diff the
meta data from a historical UNIX filesystem.
Thanks to Heiko Eissfeldt for reporting
- star: The incremental backup/restore tests now use.
"star -diff diffopts=!dnlink ..." to make them work on Cygwin.
Thanks to Heiko Eissfeldt for reporting
- star: star -c -H exustar -acl -xattr-linux .
did cause file not found messages from the attempt to archive the
Linux xattrs. This was caused by a change from July 2018 when trying to
optimize directory access in (non-find) create mode. The call to read
the Linux xattrs has now been moved to the location where in former
times the ACL code has already been moved.
The move is needed because there is no ACL/XATTR related function
that is similar to openat().
- star: A similar problem with get_xattr() did exist with star -diff
- star: star -diff did not compare ACLs since getinfo() no longer
includes a call to get_acl(). We now call get_xattr() and get_acl()
in diff.c
- star: The Solaris ACL interface now implelements lacl() / lacl_get() /
lacl_set() to support very long pathnames with ACLs.
- star: The Linux ACL interface now implelements lacl_get_file() /
lacl_set_file() to support very long pathnames with ACLs on platforms
that implement the withdrawn POSIX ACL draft.
- star: The Linux xattr interface now implelements.
llgetxattr() / llsetxattr() / lllistxattr() to support very long
pathnames with Linux xattrs.
- star: New version date
Release 2019-07-22:
- star: "pax -pe" no longer sets the variable "doxattr" as this caused
an error message:
"NFSv4 extended attribute files are not yet supported.\n"
that caused the OpenSolaris-ON "nightly" compilation to abort as a
result of that error and exit code != 0.
Release 2019-08-13:
- libschily: Various functions and *at() emulation functions call stat()
even though the caller does not know about that call.
This could cause a missbehavior in case that a file returns EOVERFLOW
with a normal stat(). We now compile these functions in unconditional
large file mode to overcome that problem.
The affected files are:
diropen.c lutimens.c findinpath.c linkat.c mkdirs.c searchinpath.c
resolvepath.c lchmod.c renameat.c.
Note that this problem affected star(1) on platforms that do not
fully implement all *at() interfaces, since star started to support
really long path names in July 2018.
- star: Star did not compile on platforms without ACLs anymore since
we did rearrange the code with schilytools 2019-07-15.
We now have the needed #undef USE_ACL in diff.c as well.
Thanks to Dennis Clarke for reporting.
- star: Avoid a warning when a star -dump archive is unpacked on FreeBSD
or Linux with non-contiguous minor bits. The warning is not needed
since we only use SCHILY.dev in order to detect mount points but
not to compute the major/minor parts.
Release 2019-08-13:
- star: star -xdev -find typically works to exclude mounted files.
It still does not always do what is expected, e.g. in case that /proc
is in the tree of scanned files, where files deep in the new mounted
tree suddenly have the same FS ID as other filesystems, e.g. the file
/proc/<pid>/path/a.out.
In such cases, "star -find -xdev" is still recommended where the
mounted file exclusion is done inside libfind instead of being
done inside star.
Before, files on other filesystems have not been honored at all when
using "star -xdev -find ...".
- improvements for Android APK and JAR archives
- better support for non-recursive list and extract
- tar --exclude-vcs support
- fixes for file attributes and flags handling
- zipx support
- rar 5.0 reader
0.12.0:
Backwards Compatibility Notes
* Support for Python 3.4 has been dropped since Python 3.4 is no longer
a supported Python version upstream. (But it will likely continue to
work until Python 2.7 support is dropped and we port to Python 3.5+
APIs.)
Bug Fixes
* Fix ``ZstdDecompressor.__init__`` on 64-bit big-endian systems.
* Fix memory leak in ``ZstdDecompressionReader.seek()``.
Changes
* CI transitioned to Azure Pipelines (from AppVeyor and Travis CI).
* Switched to ``pytest`` for running tests (from ``nose``).
* Bundled zstandard library upgraded from 1.3.8 to 1.4.3.
Version 3.1:
This will be last version with support for Python 2.x
New feature:
Accept pathlib objects as filenames.
Accept bytes filenames in Python 3
Fixes:
Use bug-compatible SHA1 for longer passwords (> 28 chars) in RAR3 encrypted headers.
Return true/false from _check_unrar_tool
Include all test files in archive
Include volume number in NeedFirstVolume exception if available (rar5).
Cleanups:
Convert tests to pytest.
v2.2.1:
Update the bundled LZ4 library to version 1.9.1
This release updates the bundled LZ4 library to version 1.9.1.
The 2.2.x releases will be the final release that support Python 2.7. In the near future we'll begin work on the 3.0.x release which will only support Python >= 3.5, and will require LZ4 > 1.9.0.
v2.2.0:
Add more detail to the install section of docs
v0.6.0
When adding implicit dirs, ensure that ancestral directories
are added and that duplicates are excluded.
The library now relies on more_itertools
5.61.0
KTar::openArchive: Don't assert if file has two root dirs
KZip::openArchive: Don't assert when opening broken files
5.60.0
Do not crash if the inner file wants to be bigger than QByteArray max size
5.59.0
Test reading and seeking in KCompressionDevice
KCompressionDevice: Remove bIgnoreData
KAr: fix out-of-bounds read (on invalid input) by porting to QByteArray
KAr: fix parsing of long filenames with Qt-5.10
KAr: the permissions are in octal, not decimal
KAr::openArchive: Also check ar_longnamesIndex is not < 0
KAr::openArchive: Fix invalid memory access on broken files
KAr::openArchive: Protect against Heap-buffer-overflow in broken files
KTar::KTarPrivate::readLonglink: Fix crash in malformed files
5.58.0
KTar: Protect against negative longlink sizes
Fix invalid memory write on malformed tar files
Fix memory leak when reading some tar files
Fix uninitialized memory use when reading malformed tar files
Fix stack-buffer-overflow read on malformed files
Fix null-dereference on malformed tar files
Install krcc.h header
Fix double delete on broken files
Disallow copy of KArchiveDirectoryPrivate and KArchivePrivate
Fix KArchive::findOrCreate running out of stack on VERY LONG paths
Introduce and use KArchiveDirectory::addEntryV2
removeEntry can fail so it's good to know if it did
KZip: fix Heap-use-after-free in broken files
LZ4 v1.9.2
fix : out-of-bound read in exceptional circumstances when using decompress_partial()
fix : slim opportunity for out-of-bound write with compress_fast() with a large enough input and when providing an output smaller than recommended (< LZ4_compressBound(inputSize))
fix : rare data corruption bug with LZ4_compress_destSize()
fix : data corruption bug when Streaming with an Attached Dict in HC Mode
perf: enable LZ4_FAST_DEC_LOOP on aarch64/GCC by default
perf: improved lz4frame streaming API speed
perf: speed up lz4hc on slow patterns when using external dictionary
api: better in-place decompression and compression support
cli : --list supports multi-frames files
cli: --version outputs to stdout
cli : add option --best as an alias of -12
misc: Integration into oss-fuzz
Zstandard v1.4.3
Dictionary Compression Regression
We discovered an issue in the v1.4.2 release, which can degrade the effectiveness of dictionary compression. This release fixes that issue.
Detailed Changes
* bug: Fix Dictionary Compression Ratio Regression
* bug: Fix Buffer Overflow in v0.3 Decompression
* build: Add support for IAR C/C++ Compiler for Arm
* misc: Add NULL pointer check in util.c by
The canonical form [1] of an R package Makefile includes the
following:
- The first stanza includes R_PKGNAME, R_PKGVER, PKGREVISION (as
needed), and CATEGORIES.
- HOMEPAGE is not present but defined in math/R/Makefile.extension to
refer to the CRAN web page describing the package. Other relevant
web pages are often linked from there via the URL field.
This updates all current R packages to this form, which will make
regular updates _much_ easier, especially using pkgtools/R2pkg.
[1] http://mail-index.netbsd.org/tech-pkg/2019/08/02/msg021711.html
2019-02-18 Stuart Caie <kyzer@cabextract.org.uk>
* chmd_read_headers(): a CHM file name beginning "::" but shorter
than 33 bytes will lead to reading past the freshly-allocated name
buffer - checks for specific control filenames didn't take length
into account. Thanks to ADLab of Venustech for the report and
proof of concept.
2019-02-18 Stuart Caie <kyzer@cabextract.org.uk>
* chmd_read_headers(): CHM files can declare their chunks are any
size up to 4GB, and libmspack will attempt to allocate that to
read the file.
This is not a security issue; libmspack doesn't promise how much
memory it'll use to unpack files. You can set your own limits by
returning NULL in a custom mspack_system.alloc() implementation.
However, it would be good to validate chunk size further. With no
offical specification, only empirical data is available. All files
created by hhc.exe have a chunk size of 4096 bytes, and this is
matched by all the files I've found in the wild, except for one
which has a chunk size of 8192 bytes, which was created by someone
developing a CHM file creator 15 years ago, and they appear to
have abandoned it, so it seems 4096 is a de-facto standard.
I've changed the "chunk size is not a power of two" warning to
"chunk size is not 4096", and now only allow chunk sizes between
22 and 8192 bytes. If you have CHM files with a larger chunk size,
please send them to me and I'll increase this upper limit.
Thanks to ADLab of Venustech for the report.
2019-02-18 Stuart Caie <kyzer@cabextract.org.uk>
* oabd.c: replaced one-shot copying of uncompressed blocks (which
requires allocating a buffer of the size declared in the header,
which can be 4GB) with a fixed-size buffer. The buffer size is
user-controllable with the new msoab_decompressor::set_param()
method (check you have version 2 of the OAB decompressor), and
also controls the input buffer used for OAB's LZX decompression.
Reminder: compression formats can dictate how much memory is
needed to decompress them. If memory usage is a security concern
to you, write a custom mspack_system.alloc() that returns NULL
if "too much" memory is requested. Do not rely on libmspack adding
special heuristics to know not to request "too much".
Thanks to ADLab of Venustech for the report.
Zstandard v1.4.2
Legacy Decompression Fix
This release is a small one, that corrects an issue discovered in the previous release. Zstandard v1.4.1 included a bug in decompressing v0.5 legacy frames, which is fixed in v1.4.2.
Detailed Changes
bug: Fix bug in zstd-0.5 decoder
bug: Fix seekable decompression in-memory API
bug: Close minor memory leak in CLI
misc: Validate blocks are smaller than size limit
misc: Restructure source files
1.0.8 (13 Jul 19)
~~~~~~~~~~~~~~~~~
* Accept as many selectors as the file format allows.
This relaxes the fix for CVE-2019-12900 from 1.0.7
so that bzip2 allows decompression of bz2 files that
use (too) many selectors again.
* Fix handling of large (> 4GB) files on Windows.
* Cleanup of bzdiff and bzgrep scripts so they don't use
any bash extensions and handle multiple archives correctly.
* There is now a bz2-files testsuite at
https://sourceware.org/git/bzip2-tests.git
1.0.7 (27 Jun 19)
~~~~~~~~~~~~~~~~~
* Fix undefined behavior in the macros SET_BH, CLEAR_BH, & ISSET_BH
* bzip2: Fix return value when combining --test,-t and -q.
* bzip2recover: Fix buffer overflow for large argv[0]
* bzip2recover: Fix use after free issue with outFile (CVE-2016-3189)
* Make sure nSelectors is not out of range (CVE-2019-12900)
v1.4.1
bug: Fix data corruption in niche use cases by @terrelln (#1659)
bug: Fuzz legacy modes, fix uncovered bugs by @terrelln (#1593, #1594, #1595)
bug: Fix out of bounds read by @terrelln (#1590)
perf: Improve decode speed by ~7% @mgrice (#1668)
perf: Slightly improved compression ratio of level 3 and 4 (ZSTD_dfast) by @cyan4973 (#1681)
perf: Slightly faster compression speed when re-using a context by @cyan4973 (#1658)
perf: Improve compression ratio for small windowLog by @cyan4973 (#1624)
perf: Faster compression speed in high compression mode for repetitive data by @terrelln (#1635)
api: Add parameter to generate smaller dictionaries by @tyler-tran (#1656)
cli: Recognize symlinks when built in C99 mode by @felixhandte (#1640)
cli: Expose cpu load indicator for each file on -vv mode by @ephiepark (#1631)
cli: Restrict read permissions on destination files by @chungy (#1644)
cli: zstdgrep: handle -f flag by @felixhandte (#1618)
cli: zstdcat: follow symlinks by @vejnar (#1604)
doc: Remove extra size limit on compressed blocks by @felixhandte (#1689)
doc: Fix typo by @yk-tanigawa (#1633)
doc: Improve documentation on streaming buffer sizes by @cyan4973 (#1629)
build: CMake: support building with LZ4 @leeyoung624 (#1626)
build: CMake: install zstdless and zstdgrep by @leeyoung624 (#1647)
build: CMake: respect existing uninstall target by @j301scott (#1619)
build: Make: skip multithread tests when built without support by @michaelforney (#1620)
build: Make: Fix examples/ test target by @sjnam (#1603)
build: Meson: rename options out of deprecated namespace by @lzutao (#1665)
build: Meson: fix build by @lzutao (#1602)
build: Visual Studio: don't export symbols in static lib by @scharan (#1650)
build: Visual Studio: fix linking by @absotively (#1639)
build: Fix MinGW-W64 build by @myzhang1029 (#1600)
misc: Expand decodecorpus coverage by @ephiepark (#1664)
Update ruby-zip to 1.2.3, here is release note.
1.2.3 (2019-05-23)
* Allow tilde in zip entry names #391 (fixes regression in 1.2.2 from #376)
* Support frozen string literals in more files #390
* Require pathname explicitly #388 (fixes regression in 1.2.2 from #376)
Tooling / Documentation:
* CI updates #392, #394
- Bump supported ruby versions and add 2.6
- JRuby failures are no longer ignored (reverts #375 / part of #371)
* Add changelog entry that was missing for last release #387
* Comment cleanup #385
Since the GitHub release information for 1.2.2 is missing, I will also include
it here:
1.2.2 (2018-09-01)
NB: This release drops support for extracting symlinks, because there was no
clear way to support this securely. See #376 (comment) for details.
* Fix CVE-2018-1000544 #376 / #371
* Fix NoMethodError: undefined method `glob' #363
* Fix handling of stored files (i.e. files not using compression) with general
purpose bit 3 set #358
* Fix close on StringIO-backed zip file #353
* Add Zip.force_entry_names_encoding option #340
* Update rubocop, apply auto-fixes, and fix regressions caused by said
auto-fixes #332, #355
* Save temporary files to temporary directory (rather than current directory)
#325
Tooling / Documentation:
* Turn off all terminal output in all tests #361
* Several CI updates #346, #347, #350, #352
* Several README improvements #345, #326, #321
### engrampa 1.22.1
sync with transifex
Help: replace link linkend with xref linkend
file-utils: avoid out of bound memory access
actions: avoid use of memory after it is freed
fr-process: Fix memory leak: 'g_shell_quote' needs to be freed
fr-process: Fix memory leak: 'g_strconcat' needs to be freed
[Security] fr-process: avoid 'strcpy' and 'strcat'
fr-process: Fix memory leak
Help: Fix version to 1.22
help: update copyright
Upgrade the manual to docbook 5.0
v1.4.0
perf: Improve level 1 compression speed in most scenarios by 6% by @gbtucker and @terrelln
api: Move the advanced API, including all functions in the staging section, to the stable section
api: Make ZSTD_e_flush and ZSTD_e_end block for maximum forward progress
api: Rename ZSTD_CCtxParam_getParameter to ZSTD_CCtxParams_getParameter
api: Rename ZSTD_CCtxParam_setParameter to ZSTD_CCtxParams_setParameter
api: Don't export ZSTDMT functions from the shared library by default
api: Require ZSTD_MULTITHREAD to be defined to use ZSTDMT
api: Add ZSTD_decompressBound() to provide an upper bound on decompressed size by @shakeelrao
api: Fix ZSTD_decompressDCtx() corner cases with a dictionary
api: Move ZSTD_getDictID_*() functions to the stable section
api: Add ZSTD_c_literalCompressionMode flag to enable or disable literal compression by @terrelln
api: Allow compression parameters to be set when a dictionary is used
api: Allow setting parameters before or after ZSTD_CCtx_loadDictionary() is called
api: Fix ZSTD_estimateCStreamSize_usingCCtxParams()
api: Setting ZSTD_d_maxWindowLog to 0 means use the default
cli: Ensure that a dictionary is not used to compress itself by @shakeelrao
cli: Add --[no-]compress-literals flag to enable or disable literal compression
doc: Update the examples to use the advanced API
doc: Explain how to transition from old streaming functions to the advanced API in the header
build: Improve the Windows release packages
build: Improve CMake build by @hjmjohnson
build: Build fixes for FreeBSD by @lwhsu
build: Remove redundant warnings by @thatsafunnyname
build: Fix tests on OpenBSD by @bket
build: Extend fuzzer build system to work with the new clang engine
build: CMake now creates the libzstd.so.1 symlink
build: Improve Menson build by @lzutao
misc: Fix symbolic link detection on FreeBSD
misc: Use physical core count for -T0 on FreeBSD by @cemeyer
misc: Fix zstd --list on truncated files by @kostmo
misc: Improve logging in debug mode by @felixhandte
misc: Add CirrusCI tests by @lwhsu
misc: Optimize dictionary memory usage in corner cases
misc: Improve the dictionary builder on small or homogeneous data
misc: Fix spelling across the repo by @jsoref
LZ4 v1.9.1
Changes
fix : decompression functions were reading a few bytes beyond input size
api : fix : lz4frame initializers compatibility with c++
cli : added command --list
build: improved Windows build
build: AIX, by Norman Green
LZ4 v1.9.0
This release brings an assortment of small improvements and bug fixes, as detailed below :
perf: large decompression speed improvement on x86/x64 (up to +20%)
api : changed : _destSize() compression variants are promoted to stable API
api : new : LZ4_initStream(HC), replacing LZ4_resetStream(HC)
api : changed : LZ4_resetStream(HC) as recommended reset function, for better performance on small data
cli : support custom block sizes
build: source code can be amalgamated, by Bing Xu
build: added meson build
build: new build macros : LZ4_DISTANCE_MAX, LZ4_FAST_DEC_LOOP
install: MidnightBSD
install: msys2 on Windows 10
Libaec provides fast lossless compression of 1 up to 32 bit wide signed
or unsigned integers (samples). The library achieves best results for
low entropy data as often encountered in space imaging instrument data or
numerical model output from weather or climate simulations. While floating
point representations are not directly supported, they can also be efficiently
coded by grouping exponents and mantissa.
Libaec implements Golomb-Rice coding as defined in the Space Data System
Standard documents 121.0-B-2 and 120.0-G-2.
Libaec includes a free drop-in replacement for the SZIP library.
Upstream changes:
0.0946 2019-04-05 20:11:47Z
- Added copyright holder/year meta to dist.ini. (GH#6) (Mohammad S Anwar)
- Auto generate META.yml using the plugin [MetaYAML]. (GH#8) (Mohammad S
Anwar)
libarchive 3.3.3:
Avoid super-linear slowdown on malformed mtree files
Many fixes for building with Visual Studio
NO_OVERWRITE doesn't change existing directory attributes
New support for Zstandard read and write filters
Update provided by Michael Bäuerle via pkgsrc-wip.
Changelog
=========
Release 2018-11-22:
-libschily: resolvenpath() did not work as expected when some path names
do not exist. A stat() call that should check whether we already
reached the "/" directory caused a return (-1) even with
(flags & RSPF_EXIST) == 0
This bug caused star to classify more symlinks as dangerous than
needed.
- star: A typo in the function dolchmodat() has been fixed. The bug has been
introduced in July 2018 while adding support for very long path names.
- star: added a new timestamp to the star version.
- star: The man page now mentions incremental backups and restores in the
FEATURES section.
Release 2018-12-06:
- star: hole.c: A memory leak in in hole.c::put_sparse() has been fixed.
Thanks to Pavel Raiskup for reporting this coverity result.
- star: xheader.c: the macro scopy() no longer has a semicolon at the end.
Thanks to Pavel Raiskup for reporting this coverity result.
Release 2019-01-22:
- libstrar & star unicode.c: iconv() may return > 0 if there are
characters that could not be converted into an
identical meaning.
We therefore now check for ret != 0 instead of
ret == -1.
- star: added support for auto detection of "zstd" compressed archives.
- star: added a new option -zstd to support compression and uncompression
using the program "zstd".
- star: Recently, star did hang in the FIFO code on Solaris. This did
not happen on Solaris over 20 years before...
On Linux - on fast multi CPU machines - the probability that a
child process from fork() starts up before the parent is 1000x higher
than on Solaris, where 10 million tries were needed to reproduce the
same problem.
As a result, the FIFO in star on Linux could in rare cases (1 of.
~ 10000 tries) even finish the 1st read() from the input file before
the "tar"-process starts with e.g. command lines like "star -tv" or.
"star -x". Since star introduced auto-byte-order detection and
handling in 1985, star needs a special start up sequence to do that.
Star introduced the FIFO in the late 1980s and the machines from that
time did always restart the parent before the fork()ed child starts.
The new OS behavior thus caused a situation that was not forseeable
when the FIFO has been designed. This new OS behavior caused a
deadlock in aprox. 1 of 10000 star calls on Linux and 1 of 10000000
star calls on Solaris.
Star now waits when entering the FIFO fill-process until the.
FIFO get-process did start up before trying to wake up a waiting
get process.
- star: On Linux, in 1 of 1.5 million tries, star did die from SIGPIPE.
Note that this did never happen on Solaris.
Star now ignores SIGPIPE and it seems that this fixed the problem
since it did not happen again after that change with even 100 million
tries.
- star: The debug printing for the FIFO has been enhanced to print more
information from the FIFO control structure to make it easier to debug
problems like the ones mentioned above.
- star: There seems to be a problem in pipe handling in the Linux kernel.
It seems that in rare cases, the read(2) on a pipe returns 0 even though
the write side did write(2) one byte to the pipe just before calling
exit(). Unfortunately, this problem is hard to debug as it happens only
once every ~30 million tries. Our workaround is to behave as if the
expected byte could be read and star currently prints something like:
star: Erfolg. Sync pipe read error pid 8141 ret 0
star: Erfolg. Ib 0 Ob 1 e 0 p 1 g 0 chan 5.
star: Erfolg. Trying to work around Kernel Pipe botch.
before it continues. Since the star exit code in such a case is 0,
we assume that this is a correct workaround and this case thus may
be made completely silent in the future.
- star: an even less frequent FIFO problem (occurs once every 50 million
tries on fast multi CPU machines) has been identified. Star reports a
hard EOF on input even though the complete file with logical EOF has
been read and there is still input to process. In order to debug this
problem a debug message has been added to the code.
With this debug message, it turned out, that this problem happened
because a context switch occurred in the FIFO read process after it did
see an empty FIFO and later, after the process was resumed, the
following check for the FIFO_MEOF flag did see EOF. We now first check
for the FIFO_MEOF flag and later for the amount of data inside as the
FIFO as FIFO_MEOF is set after the FIFO content has been updated and
thus a context switch is no longer able to cause a wrong assumption
about the content of the FIFO.
If you still see this, please send a report.
- star: added support to print debug malloc statistics to better debug
memory problems in star.
- star: pathname.c:: free_pspace() now only frees the path buffer if it
is != NULL
- star: fixed a bug in the file create.c that caused star to incorrectly
grow the path buffer by 2 bytes for every archived file. This caused
star to constantly grow if a larger amount of files are archived and
eat up all memory available to 32 bit processes if the archived
filesystem is larger than approx. 1 TB.
- star: If the path name now cannot be handled because of low memory,
we print a warning that includes the text "out of memory".
- star: Now checking whether open of /dev/null failed while running a
compress pipe. This avoids a core dump on defective OS installations.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: props.c: Added a missing /* FALLTHROUGH */ comment..
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: create.c: Add more comment for the CPIO CRC format handler to
explain why the last instance if a series of hard links for a file
needs to archive the data.
- star: diff.c: added a filling fillbytes(&finfo, ...) to make sure that
ACL pointers are initialized.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: Several /* NOTREACHED */ comments have been added to tell
programs like coverity that after a NULL pointer check, there is no
continuation of the program
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: extract.c: A if (path->ps_path == '\0') has been corrected to
if (path->ps_path[0] == '\0') after a mktemp() call. This was a typo
introduced with the new support for extremely long path names.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: extract.c An initalization for a struct pathstore has been
moved to the front to verify that path.ps_path is always initialized.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: header.c: isgnumagic(&ptb->dbuf.t_vers) has been changed to
isgnumagic(ptb->ustar_dbuf.t_magic) as it is a "ustar" structure
that is going to be checked.
Thanks to Pavel Raiskup for poiting to a related Coverity message.
- star: some Cstyle changes
- bsh / Bourne Shell / star: the function hop_dirs() no longer checks
for p2 != NULL before calling *p2 = '/' as p2 has
been granted to be != NULL from a break with
strchr(p, '/') == NULL
Release 2019-02-18:
- star: another similar has been fixed similat to what has been fixed
already in the 2019-01-22 release:
An even less frequent FIFO problem (occurs once every 50 million
tries on fast multi CPU machines) has been identified. Star reports a
hard EOF on input even though the complete file with logical EOF has
been read and there is still input to process. In order to debug this
problem a debug message has been added to the code.
With this debug message, it turned out, that this problem happened
because a context switch occurred in the FIFO read process after it did
see an empty FIFO and later, after the process was resumed, the
following check for the FIFO_MEOF flag did see EOF. We now first check
for the FIFO_MEOF flag and later for the amount of data inside as the
FIFO as FIFO_MEOF is set after the FIFO content has been updated and
thus a context switch is no longer able to cause a wrong assumption
about the content of the FIFO.
We now did run 250 million tests without seeing another problem.
If you still see this, please send a report.
- star: Note that the debug output for this problem now has been
disabled. If you need to debug this, call:
smake clean COPTX=-DFIFO_EOF_DEBUG all
in the star directory.
- star: The message "Sync pipe read error" is no longer printed when
the FIFO background process dies instead of sending a final wakeup.
This is needed since there is a possibility for a context switch in
the foreground process that can make it later wait for a wakeup while
the background process misses to see the wait flag and just exits.
- star: In rare conditions (once every 2 million tries), a hang could.
occur with "star -c" if the tar process fills the FIFO and sets the
EOF flag and then calls wait() to wait for the FIFO tape output
process. This happens in case that the tape output did not see the
EOF flag because it has undergone a context switch after it checked
for the not yet existing EOF flag and before waiting for a wakeup
from the tar FIFO fill process.
Star now closes the sync pipes before calling wait() as this always
wakes up the waiting other side.
We did run another 300 million tests for this condition and did not
see any problem now.
- star: The version is now 1.6
Short overview for what changed since the last "stable" version:
- Support for "infinitely" long path names has been added.
- Support for comparing timestamps with nanosecond granularity
- -secure-links has been made the default when extracting
archived (except when doing an incremental restore).
- Added Support for NFSv4 ACLs on FreeBSD. Solaris has been
supported since 2013.
- Added Support to archive SELinix attributes.
- Allow to configure whether "star -fsync" is the default in
order to support filesystems that are slow with granted
transactions (like ZFS) or platforms that are genrally
slow with fsync() (like Linux).
- Full UNICODE support has been added for tar headers.
- Support for -zstd compression has been added.
- Some rare FIFO problems have been fixed.
Note that we did recently run more than a billion tests to
verify the FIFO after we identified a method to trigger the
problem on Linux.
Release 2019-03-11:
- star: Support for base-256 numbers in timestams and uid/gid has been
added. This has been planned in the 1990s already, when star invented
the base-256 coding, but it has been forgotten in favor of the
POSIX.1-2001 enhanded archive headers. Now it seems that GNU tar.
that copied the format from star uses it for timestamps and uid/gid
and we need to implement it in order to get archive compatibility.
Thanks to Michal Górny (mgorny@gentoo.org) for detecting the missing
feature.
- star: The t_rdev field in the old star header now may use base-256
as well.
- star: The function stoli() added a new parameter "fieldwidth" that
allows to configure when a "unterminated octal number" warning is
printed. This is needed since this function is used for 8 byte and
for 12 byte fields.
- star: star did print archives with illegal 32 byte user/group.
names (where the nul terminator is missing) "correctly", when in.
list mode but it used only the first 31 bytes when extractig.
such archives
- star: a new function istarnumber() is used to do better heuristics on
what a valid TAR archive is. We have some special handling to work.
around the non-compliance of GNU tar in some known cases. If you
discover other GNU tar archives that are not detected as TAR archive,
please report them to help to make th eheuristics better.
The background is to make star better in detecting fool archives.
- star: The directory testscripts added new files:
testscripts/not_a_tar_file1 and testscripts/not_a_tar_file3
with correct checksums that fool tar implementations that use too
few heuristics to identify tar archives.
- star: fixed a bug in the FIFO related to extracting multi-volume
archives. The bug was introduced with release 2019-02-18 and the
effect was that the FIFO complained at the end of the last volume.
- star/libschily: Added new error checking codes:
"ID"<-->allows to control error behaviour with range errors in uid_t
and gid_t values.
"TIME"<>allows to control error behaviour with range errors in time_t
- star: Creating multi volume archives without using the FIFO did dump
core. We thus no longer set mp->chreel = TRUE; when the FIFO has.
been disabled. The related bug has been introduced in January 2012.
- star: Creating multi volume archives with a very small volume size
could cause a hang at the end as the function startvol() did not
check whether the TAR process did already decide to exit while
waiting for the TAR process to calm down (stop) before writing the
next multi volume header. We no longer wait in this case.
- star: exprstats() now calls fifo_exit(ret) in order to avoid a
FIFO Sync pipe read error message in case that star was terminated
with an error.
- star: Since we added better Unicode support in May 2018, star did
dump core when a multi volume header with POSIX.1-2001 extensions
was written in multi volume create mode. We now check for NULL
pointers before we call nameascii() to decide whether the file.
name needs a UTF-8 translation.
- star: Creating multi volume archives without POSIX.1-2001 support
no longer sets POSIX.1-2001 extension flags for the volume header.
- star: The flag XF_NOTIME now works when creating POSIX.1-2001
extended headers and thus the 'x'-header with time stamps for the
volume header tar header is no longer created. This avoids
to write atime=1 for the volume number 1 since we encode the
volume number in the otherwise useless atime of the volume header
when in POSIX.1-1988 TAR mode.
- star: the star.1 man page now mentions that the first tar program
appeared in 1979 (3 years before star has been started as a project).
- star: the star.4 man page now has a "SEE ALSO", a HISTORY and
a AUTHOR section.
- star: the star.4 man page now has a MULTI VOLUME ARCHIVE HANDLING
section.
- star: the star.4 man page added a new "BASIC TAR STRUCTURE" section.
- star: The ACL reference test archives (formerly available from e.g.:
http://sf.net/projects/s-tar/files/alpha/) have been added
to the directory star/testscripts/. The files.
acl-test.tar.gz
acl-test2.tar.gz
acl-test3.tar.gz
acl-test4.tar.gz
acl-test5.tar.gz
contain ACLs that use the obsolete method from a POSIX proposal
from around 1993 that was withdrawn in 1997 and never has become
part of a standard. This method has been implemented in 1993 for
UFS on Solaris.
GNU tar claims to support this format but really does not support
it at all. GNU tar fails to extract the reference tar archives from
above and it fails to create a compliant tar archive in create mode.
It is strange to see that GNU tar never has been tested against the
reference archives that have been created in collaboration with
SuSE in 2001 already.
The files
acl-nfsv4-test.tar.gz
acl-nfsv4-test2.tar.gz
acl-nfsv4-test3.tar.gz
acl-nfsv4-test4.tar.gz
acl-nfsv4-test5.tar.gz
contain ACLs that have become part of the NFSv4 standard and that.
are also used on NTFS and ZFS. This format is completely unsupported
by GNU tar.
- star TODO: create unit tests in order to avoid future problems
with multi volume archives similar to the problems we recently
fixed.
- star: Updated version 1.6 (not yet published in separate tarball)
Short overview for what changed since the last "stable" version:
- Support for "infinitely" long path names has been added.
- Support for base-256 numbers in timestams and uid/gid
has been added. This has been planned in the 1990s already,
when star invented the base-256 coding, but it has been
forgotten in favor of the POSIX.1-2001 enhanded archive
headers.
- Support for comparing timestamps with nanosecond granularity
- -secure-links has been made the default when extracting
archived (except when doing an incremental restore).
- Added Support for NFSv4 ACLs on FreeBSD. Solaris has been
supported since 2013.
- Added Support to archive SELinix attributes.
- Allow to configure whether "star -fsync" is the default in
order to support filesystems that are slow with granted
transactions (like ZFS) or platforms that are genrally
slow with fsync() (like Linux).
- Full UNICODE support has been added for tar headers.
- Support for -zstd compression has been added.
- Some rare FIFO problems have been fixed.
Note that we did recently run more than a billion tests to
verify the FIFO after we identified a method to trigger the
problem on Linux.
1.5.2 [2019-03-12]
==================
* Fix bug in AES encryption affecting certain file sizes
* Keep file permissions when modifying zip archives
* Support systems with small stack size.
* Support mbed TLS as crypto backend.
* Add nullability annotations.
### engrampa 1.22.0
* Translations update
* Avoid array index out of bounds parsing dpkg-deb --info
* warning: Use of memory after it is freed
* Read authors (updated) from engrampa.about gresource
* Enable Travis CI
* eggsmclient: avoid deprecated 'g_type_class_add_private'
* update copyright year to 2019
* rar/unrar: Fix: "overwrite existing files" disabled must work
* fix fr-command-cfile.c: fr_process_set_working_dir
* fr-command-cfile.c: fix indentation
* Added test integrity for brotli
* Added test integrity for the cfile compressors: gzip, bzip2, etc.
* move appdata to metainfo directory
* fr-window: show the pause button only if the dialog is working
* disable deprecation warnings for distcheck
* fr-window: avoid 'gtk_dialog_add_button' with stock ids
* fr-window: hide the progress bar if the process is paused
* fr-window: change the info label if process is paused/resumed
* fr-window: little improvements in the look of pause/resume button
* Adding pause and start functions
* Fix implementation and use of the alternative package name lookup
* Added support for brotli (*.tar.br) compressed tar archives
* Add brotli support
* Use make functions for HELP_LINGUAS
* Replace -Dokumentationteam
* Replace -Dokumentationsprojekt with Documentation Project
* Manual: Update file format descriptions using shared-mime-info
* Fix url of ulinks to point to mate-user-guide
* UNIX and Linux systems -> Linux and UNIX-like systems
* tx: add atril help to transifex config
* Add the ability to support 'unar' over .zip archives
* Add support for OpenDocument formats
* UI: on the properties dialog, focus the Close button instead of the Help button by default
0.11.0 (released 2019-02-24)
============================
Backwards Compatibility Nodes
-----------------------------
* ZstdDecompressor.read() now allows reading sizes of -1 or 0
and defaults to -1, per the documented behavior of
io.RawIOBase.read(). Previously, we required an argument that was
a positive value.
* The readline(), readlines(), __iter__, and __next__ methods
of ZstdDecompressionReader() now raise io.UnsupportedOperation
instead of NotImplementedError.
* ZstdDecompressor.stream_reader() now accepts a read_across_frames
argument. The default value will likely be changed in a future release
and consumers are advised to pass the argument to avoid unwanted change
of behavior in the future.
* setup.py now always disables the CFFI backend if the installed
CFFI package does not meet the minimum version requirements. Before, it was
possible for the CFFI backend to be generated and a run-time error to
occur.
* In the CFFI backend, CompressionReader and DecompressionReader
were renamed to ZstdCompressionReader and ZstdDecompressionReader,
respectively so naming is identical to the C extension. This should have
no meaningful end-user impact, as instances aren't meant to be
constructed directly.
* ZstdDecompressor.stream_writer() now accepts a write_return_read
argument to control whether write() returns the number of bytes
read from the source / written to the decompressor. It defaults to off,
which preserves the existing behavior of returning the number of bytes
emitted from the decompressor. The default will change in a future release
so behavior aligns with the specified behavior of io.RawIOBase.
* ZstdDecompressionWriter.__exit__ now calls self.close(). This
will result in that stream plus the underlying stream being closed as
well. If this behavior is not desirable, do not use instances as
context managers.
* ZstdCompressor.stream_writer() now accepts a write_return_read
argument to control whether write() returns the number of bytes read
from the source / written to the compressor. It defaults to off, which
preserves the existing behavior of returning the number of bytes emitted
from the compressor. The default will change in a future release so
behavior aligns with the specified behavior of io.RawIOBase.
* ZstdCompressionWriter.__exit__ now calls self.close(). This will
result in that stream plus any underlying stream being closed as well. If
this behavior is not desirable, do not use instances as context managers.
* ZstdDecompressionWriter no longer requires being used as a context
manager.
* ZstdCompressionWriter no longer requires being used as a context
manager.
* The overlap_size_log attribute on CompressionParameters instances
has been deprecated and will be removed in a future release. The
overlap_log attribute should be used instead.
* The overlap_size_log argument to CompressionParameters has been
deprecated and will be removed in a future release. The overlap_log
argument should be used instead.
* The ldm_hash_every_log attribute on CompressionParameters instances
has been deprecated and will be removed in a future release. The
ldm_hash_rate_log attribute should be used instead.
* The ldm_hash_every_log argument to CompressionParameters has been
deprecated and will be removed in a future release. The ldm_hash_rate_log
argument should be used instead.
* The compression_strategy argument to CompressionParameters has been
deprecated and will be removed in a future release. The strategy
argument should be used instead.
* The SEARCHLENGTH_MIN and SEARCHLENGTH_MAX constants are deprecated
and will be removed in a future release. Use MINMATCH_MIN and
MINMATCH_MAX instead.
* The zstd_cffi module has been renamed to zstandard.cffi. As had
been documented in the README file since the 0.9.0 release, the
module should not be imported directly at its new location. Instead,
import zstandard to cause an appropriate backend module to be loaded
automatically.
Bug Fixes
---------
* CFFI backend could encounter a failure when sending an empty chunk into
ZstdDecompressionObj.decompress(). The issue has been fixed.
* CFFI backend could encounter an error when calling
ZstdDecompressionReader.read() if there was data remaining in an
internal buffer. The issue has been fixed.
Changes
-------
* ZstDecompressionObj.decompress() now properly handles empty inputs in
the CFFI backend.
* ZstdCompressionReader now implements read1() and readinto1().
These are part of the io.BufferedIOBase interface.
* ZstdCompressionReader has gained a readinto(b) method for reading
compressed output into an existing buffer.
* ZstdCompressionReader.read() now defaults to size=-1 and accepts
read sizes of -1 and 0. The new behavior aligns with the documented
behavior of io.RawIOBase.
* ZstdCompressionReader now implements readall(). Previously, this
method raised NotImplementedError.
* ZstdDecompressionReader now implements read1() and readinto1().
These are part of the io.BufferedIOBase interface.
* ZstdDecompressionReader.read() now defaults to size=-1 and accepts
read sizes of -1 and 0. The new behavior aligns with the documented
behavior of io.RawIOBase.
* ZstdDecompressionReader() now implements readall(). Previously, this
method raised NotImplementedError.
* The readline(), readlines(), __iter__, and __next__ methods
of ZstdDecompressionReader() now raise io.UnsupportedOperation
instead of NotImplementedError. This reflects a decision to never
implement text-based I/O on (de)compressors and keep the low-level API
operating in the binary domain.
* README.rst now documented how to achieve linewise iteration using
an io.TextIOWrapper with a ZstdDecompressionReader.
* ZstdDecompressionReader has gained a readinto(b) method for
reading decompressed output into an existing buffer. This allows chaining
to an io.TextIOWrapper on Python 3 without using an io.BufferedReader.
* ZstdDecompressor.stream_reader() now accepts a read_across_frames
argument to control behavior when the input data has multiple zstd
*frames*. When False (the default for backwards compatibility), a
read() will stop when the end of a zstd *frame* is encountered. When
True, read() can potentially return data spanning multiple zstd
*frames*. The default will likely be changed to True in a future
release.
* setup.py now performs CFFI version sniffing and disables the CFFI
backend if CFFI is too old. Previously, we only used install_requires
to enforce the CFFI version and not all build modes would properly enforce
the minimum CFFI version.
* CFFI's ZstdDecompressionReader.read() now properly handles data
remaining in any internal buffer. Before, repeated read() could
result in *random* errors.
* Upgraded various Python packages in CI environment.
* Upgrade to hypothesis 4.5.11.
* In the CFFI backend, CompressionReader and DecompressionReader
were renamed to ZstdCompressionReader and ZstdDecompressionReader,
respectively.
* ZstdDecompressor.stream_writer() now accepts a write_return_read
argument to control whether write() returns the number of bytes read
from the source. It defaults to False to preserve backwards
compatibility.
* ZstdDecompressor.stream_writer() now implements the io.RawIOBase
interface and behaves as a proper stream object.
* ZstdCompressor.stream_writer() now accepts a write_return_read
argument to control whether write() returns the number of bytes read
from the source. It defaults to False to preserve backwards
compatibility.
* ZstdCompressionWriter now implements the io.RawIOBase interface and
behaves as a proper stream object. close() will now close the stream
and the underlying stream (if possible). __exit__ will now call
close(). Methods like writable() and fileno() are implemented.
* ZstdDecompressionWriter no longer must be used as a context manager.
* ZstdCompressionWriter no longer must be used as a context manager.
When not using as a context manager, it is important to call
flush(FRAME_FRAME) or the compression stream won't be properly
terminated and decoders may complain about malformed input.
* ZstdCompressionWriter.flush() (what is returned from
ZstdCompressor.stream_writer()) now accepts an argument controlling the
flush behavior. Its value can be one of the new constants
FLUSH_BLOCK or FLUSH_FRAME.
* ZstdDecompressionObj instances now have a flush([length=None]) method.
This provides parity with standard library equivalent types.
* CompressionParameters no longer redundantly store individual compression
parameters on each instance. Instead, compression parameters are stored inside
the underlying ZSTD_CCtx_params instance. Attributes for obtaining
parameters are now properties rather than instance variables.
* Exposed the STRATEGY_BTULTRA2 constant.
* CompressionParameters instances now expose an overlap_log attribute.
This behaves identically to the overlap_size_log attribute.
* CompressionParameters() now accepts an overlap_log argument that
behaves identically to the overlap_size_log argument. An error will be
raised if both arguments are specified.
* CompressionParameters instances now expose an ldm_hash_rate_log
attribute. This behaves identically to the ldm_hash_every_log attribute.
* CompressionParameters() now accepts a ldm_hash_rate_log argument that
behaves identically to the ldm_hash_every_log argument. An error will be
raised if both arguments are specified.
* CompressionParameters() now accepts a strategy argument that behaves
identically to the compression_strategy argument. An error will be raised
if both arguments are specified.
* The MINMATCH_MIN and MINMATCH_MAX constants were added. They are
semantically equivalent to the old SEARCHLENGTH_MIN and
SEARCHLENGTH_MAX constants.
* Bundled zstandard library upgraded from 1.3.7 to 1.3.8.
* setup.py denotes support for Python 3.7 (Python 3.7 was supported and
tested in the 0.10 release).
* zstd_cffi module has been renamed to zstandard.cffi.
* ZstdCompressor.stream_writer() now reuses a buffer in order to avoid
allocating a new buffer for every operation. This should result in faster
performance in cases where write() or flush() are being called
frequently.
* Bundled zstandard library upgraded from 1.3.6 to 1.3.7.
version 1.32 - Sergey Poznyakoff, 2019-02-23
* Fix the use of --checkpoint without explicit --checkpoint-action
* Fix extraction with the -U option
See http://lists.gnu.org/archive/html/bug-tar/2019-01/msg00015.html,
for details
* Fix iconv usage on BSD-based systems
* Fix possible NULL dereference (savannah bug #55369)
* Improve the testsuite
v2.1.5: Made the md5sum detection consistent with the header code. Check for
the presence of the archive directory. Added --encrypt for symmetric encryption
through gpg (Eric Windisch). Added support for the digest command on Solaris 10
for MD5 checksums. Check for available disk space before extracting to the
target directory (Andreas Schweitzer). Allow extraction to run asynchronously
(patch by Peter Hatch). Use file descriptors internally to avoid error messages
(patch by Kay Tiong Khoo).
v2.1.6: Replaced one dot per file progress with a realtime progress percentage
and a spining cursor. Added --noprogress to prevent showing the progress during
the decompression. Added --target dir to allow extracting directly to a target
directory. (Guy Baconniere)
v2.2.0: First major new release in years! Includes many bugfixes and user
contributions. Please look at the project page on Github for all the details.
v2.3.0: Support for archive encryption via GPG or OpenSSL. Added LZO and LZ4
compression support. Options to set the packaging date and stop the umask from
being overriden. Optionally ignore check for available disk space when
extracting. New option to check for root permissions before extracting.
v2.3.1: Various compatibility updates. Added unit tests for Travis CI in the
GitHub repo. New --tar-extra, --untar-extra, --gpg-extra,
--gpg-asymmetric-encrypt-sign options.
v2.4.0: Added optional support for SHA256 archive integrity checksums.
Changes in version 1.21:
The options '--dump', '--remove' and '--strip' have been added, mainly as
support for the tarlz archive format: http://www.nongnu.org/lzip/tarlz.html
These options replace '--dump-tdata', '--remove-tdata' and '--strip-tdata',
which are now aliases and will be removed in version 1.22.
'--dump=[<member_list>][:damaged][:tdata]' dumps the members listed, the
damaged members (if any), or the trailing data (if any) of one or more
regular multimember files to standard output.
'--remove=[<member_list>][:damaged][:tdata]' removes the members listed,
the damaged members (if any), or the trailing data (if any) from regular
multimember files in place.
'--strip=[<member_list>][:damaged][:tdata]' copies one or more regular
multimember files to standard output, stripping the members listed, the
damaged members (if any), or the trailing data (if any) from each file.
Detection of forbidden combinations of characters in trailing data has been
improved.
'--split' can now detect trailing data and gaps between members, and save
each gap in its own file. Trailing data (if any) are saved alone in the last
file. (Gaps may contain garbage or may be members with corrupt headers or
trailers).
'--ignore-errors' now makes '--list' show gaps between members, ignoring
format errors.
'--ignore-errors' now makes '--range-decompress' ignore a truncated last
member.
Errors are now also checked when closing the input file in decompression
mode.
Some diagnostic messages have been improved.
'\n' is now printed instead of '\r' when showing progress of merge or repair
if stdout is not a terminal.
Lziprecover now compiles on DOS with DJGPP. (Patch from Robert Riebisch).
The new chapter 'Tarlz', explaining the ways in which lziprecover can
recover and process multimember tar.lz archives, has been added to the
manual.
The configure script now accepts appending options to CXXFLAGS using the
syntax 'CXXFLAGS+=OPTIONS'.
It has been documented in INSTALL the use of
CXXFLAGS+='-D __USE_MINGW_ANSI_STDIO' when compiling on MinGW.
Changes in version 1.21:
Detection of forbidden combinations of characters in trailing data has been
improved.
Errors are now also checked when closing the input file.
Lzip now compiles on DOS with DJGPP. (Patch from Robert Riebisch).
The descriptions of '-0..-9', '-m' and '-s' in the manual have been
improved.
The configure script now accepts appending options to CXXFLAGS using the
syntax 'CXXFLAGS+=OPTIONS'.
It has been documented in INSTALL the use of
CXXFLAGS+='-D __USE_MINGW_ANSI_STDIO' when compiling on MinGW.
Prompted in part because prior releases fail to build on Linux
distributions that use glibc >= 2.27 (relates to PR pkg/53826).
* Noteworthy changes in release 1.10 (2018-12-29) [stable]
** Changes in behavior
Compressed gzip output no longer contains the current time as a
timestamp when the input is not a regular file. Instead, the output
contains a null (zero) timestamp. This makes gzip's behavior more
reproducible when used as part of a pipeline. (As a reminder, even
regular files will use null timestamps after the year 2106, due to a
limitation in the gzip format.)
** Bug fixes
A use of uninitialized memory on some malformed inputs has been fixed.
[bug present since the beginning]
A few theoretical race conditions in signal handers have been fixed.
These bugs most likely do not happen on practical platforms.
[bugs present since the beginning]
* Noteworthy changes in release 1.9 (2018-01-07) [stable]
** Bug fixes
gzip -d -S SUFFIX file.SUFFIX would fail for any upper-case byte in SUFFIX.
E.g., before, this command would fail:
$ :|gzip > kT && gzip -d -S T kT
gzip: kT: unknown suffix -- ignored
[bug present since the beginning]
When decompressing data in 'pack' format, gzip no longer mishandles
leading zeros in the end-of-block code. [bug introduced in gzip-1.6]
When converting from system-dependent time_t format to the 32-bit
unsigned MTIME format used in gzip files, if a timestamp does not
fit gzip now substitutes zero instead of the timestamp's low-order
32 bits, as per Internet RFC 1952. When converting from MTIME to
time_t format, if a timestamp does not fit gzip now warns and
substitutes the nearest in-range value instead of crashing or
silently substituting an implementation-defined value (typically,
the timestamp's low-order bits). This affects timestamps before
1970 and after 2106, and timestamps after 2038 on platforms with
32-bit signed time_t. [bug present since the beginning]
Commands implemented via shell scripts are now more consistent about
failure status. For example, 'gunzip --help >/dev/full' now
consistently exits with status 1 (error), instead of with status 2
(warning) on some platforms. [bug present since the beginning]
Support for VMS and Amiga has been removed. It was not working anyway,
and it reportedly caused file name glitches on MS-Windowsish platforms.
* Noteworthy changes in release 1.8 (2016-04-26) [stable]
** Bug fixes
gzip -l no longer falsely reports a write error when writing to a pipe.
[bug introduced in gzip-1.7]
Port to Oracle Solaris Studio 12 on x86-64.
[bug present since at least gzip-1.2.4]
When configuring gzip, ./configure DEFS='...-DNO_ASM...' now
suppresses assembler again. [bug introduced in gzip-1.3.5]
* Noteworthy changes in release 1.7 (2016-03-27) [stable]
** Changes in behavior
The GZIP environment variable is now obsolescent; gzip now warns if
it is used, and rejects attempts to use dangerous options or operands.
You can use an alias or script instead.
Installed programs like 'zgrep' now use the PATH environment variable
as usual to find subsidiary programs like 'gzip' and 'grep'.
Previously they prepended the installation directory to the PATH,
which sometimes caused 'make check' to test the wrong gzip executable.
[bug introduced in gzip-1.3.13]
** New features
gzip now accepts the --synchronous option, which causes it to use
fsync and similar primitives to transfer output data to the output
file's storage device when the file system supports this. Although
this option makes gzip safer in the presence of system crashes, it
can make gzip considerably slower.
gzip now accepts the --rsyncable option. This option is accepted in
all modes, but has effect only when compressing: it makes the resulting
output more amenable to efficient use of rsync. For example, when a
large input file gets a small change, a gzip --rsyncable image of
that file will remain largely unchanged, too. Without --rsyncable,
even a tiny change in the input could result in a totally different
gzip-compressed output file.
** Bug fixes
gzip -k -v no longer reports that files are replaced.
[bug present since the beginning]
zgrep -f A B C no longer reads A more than once if A is not a regular file.
This better supports invocations like 'zgrep -f <(COMMAND) B C' in Bash.
[bug introduced in gzip-1.2]
Bz2file is a Python library for reading and writing bzip2-compressed files. It
contains a drop-in replacement for the file interface in the standard library's
bz2 module, including features from the latest development version of CPython
that are not available in older releases.
Changelog:
version 1.31 - Sergey Poznyakoff, 2019-01-02
* Fix heap-buffer-overrun with --one-top-level.
Bug introduced with the addition of that option in 1.28.
* Support for zstd compression
New option '--zstd' instructs tar to use zstd as compression program.
When listing, extractng and comparing, zstd compressed archives are
recognized automatically.
When '-a' option is in effect, zstd compression is selected if the
destination archive name ends in '.zst' or '.tzst'.
* The -K option interacts properly with member names given in the command line
Names of members to extract can be specified along with the "-K NAME"
option. In this case, tar will extract NAME and those of named members
that appear in the archive after it, which is consistent with the
semantics of the option.
Previous versions of tar extracted NAME, those of named members that
appeared before it, and everything after it.
* Fix CVE-2018-20482
When creating archives with the --sparse option, previous versions of
tar would loop endlessly if a sparse file had been truncated while
being archived.
Zstandard v1.3.8
perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)
perf: slightly better compression ratio at high compression modes
api : finalized advanced API, last stage before "stable" status
api : new --rsyncable mode
api : support decompression of empty frames into NULL (used to be an error)
build: new set of build macros to generate a minimal size decoder
build: fix compilation on MIPS32
build: fix compilation with multiple -arch flags
build: highly upgraded meson build
build: improved buck support
build: fix cmake script : can create debug build
build: Makefile : grep works on both colored consoles and systems without color support
build: fixed zstd-pgo target
cli : support ZSTD_CLEVEL environment variable
cli : --no-progress flag, preserving final summary
cli : ensure destination file is not source file
cli : clearer error messages, notably when input file not present
doc : clarified zstd_compression_format.md
misc: fixed zstdgrep, returns 1 on failure
misc: NEWS renamed as CHANGELOG, in accordance with fb.oss policy
2.1.5
This release contains no functional changes other than changes to the Appveyor configuration for publishing wheels.
2.1.4
This release contains no functional changes other than changes to the Travis configuration for publishing wheels.
2.1.3
A simplification of the tox.ini file
More robust checking for pkgconfig availability
Integration of cibuildwheel into travis builds so as to build and publish binary wheels for Linux and OSX
Only require pytest-runner if pytest/test is being called
Blacklists version 3.3.0 of pytest which has a bug that can cause the tests to fail.
1.0.7
cross compilation support:
added ability to run cross-compiled ARM tests in qemu
added arm-linux-gnueabihf-gcc entry to Travis build matrix
faster decoding on ARM:
implemented prefetching HuffmanCode entry as uint32_t if target platform is ARM
fixed NEON extension detection
combed Huffman table building code for better readability
improved precision of window size calculation in CLI
minor fixes:
fixed typos
improved internal comments / parameter names
fixed BROTLI_PREDICT_TRUE/_FALSE detection for SunPro compiler
unburdened JNI (Bazel) builds from fetching the full JDK
1.0.6
Fixes
fix unaligned 64-bit accesses on AArch32
add missing files to the sources list
add ASAN/MSAN unaligned read specializations
fix CoverityScan "unused assignment" warning
fix JDK 8<->9 incompatibility
unbreak Travis builds
fix auto detect of bundled mode in cmake
* libmspack is now distributed with its test-suite, which now run
as part of "make check"
* libmspack's programs in src/ have been moved to examples/ and do
not auto-install
Set TEST_TARGET.
New in 1.9
* Fixed invisible bad extraction when using cabextract -F (broken in 1.8)
* Fixed configure --with-external-libmspack which was broken in 1.8
* configure --with-external-libmspack will now use pkg-config. To configure
it manually, set environment variables libmspack_CFLAGS and libmspack_LIBS
before running configure.
* Now includes the test suite (make check)
New in 1.8
* cabextract -f now extracts even more badly damaged files than before
Uptsream changes:
2.32 13/09/2018 (CBERRY)
- Fix absolute path handling on VMS
2.30 19/06/2018
- skip white_space test on MSWin32 as Windows will report that both
files exist, which is obviously a 'feature'
2.1.2:
Improves the speed of importing the module by avoiding the use of pkg_resources
Fixes some flake8 warnings
Resolves a small issue with the test suite when detecting memory usage increases
* On Arch Linux, the build failed, makedev(3) indicates
#include <sys/sysmacros.h>
* On Debian Buster, the build succeed but a big warning is displayed:
warning: In the GNU C Library, "minor" is defined
by <sys/sysmacros.h>. For historical compatibility, it is
currently defined by <sys/types.h> as well, but we plan to
remove this soon. To use "minor", include <sys/sysmacros.h>
directly. If you did not intend to use a system-defined macro
"minor", you should undefine it after including <sys/types.h>.
0.10.1:
Backwards Compatibility Notes
* ZstdCompressor.stream_reader().closed is now a property instead of a
method.
* ZstdDecompressor.stream_reader().closed is now a property instead of a
method.
Changes
* Stop attempting to package Python 3.6 for Miniconda. The latest version of
Miniconda is using Python 3.7. The Python 3.6 Miniconda packages were a lie
since this were built against Python 3.7.
* ZstdCompressor.stream_reader()'s and ZstdDecompressor.stream_reader()'s
closed attribute is now a read-only property instead of a method. This now
properly matches the IOBase API and allows instances to be used in more
places that accept IOBase instances.
0.10.0:
Backwards Compatibility Notes
* ZstdDecompressor.stream_reader().read() now consistently requires an
argument in both the C and CFFI backends. Before, the CFFI implementation
would assume a default value of -1, which was later rejected.
* The compress_literals argument and attribute has been removed from
zstd.ZstdCompressionParameters because it was removed by the zstd 1.3.5
API.
* ZSTD_CCtx_setParametersUsingCCtxParams() is no longer called on every
operation performed against ZstdCompressor instances. The reason for this
change is that the zstd 1.3.5 API no longer allows this without calling
ZSTD_CCtx_resetParameters() first. But if we called
ZSTD_CCtx_resetParameters() on every operation, we'd have to redo
potentially expensive setup when using dictionaries. We now call
ZSTD_CCtx_reset() on every operation and don't attempt to change
compression parameters.
* Objects returned by ZstdCompressor.stream_reader() no longer need to be
used as a context manager. The context manager interface still exists and its
behavior is unchanged.
* Objects returned by ZstdDecompressor.stream_reader() no longer need to be
used as a context manager. The context manager interface still exists and its
behavior is unchanged.
Bug Fixes
* ZstdDecompressor.decompressobj().decompress() should now return all data
from internal buffers in more scenarios. Before, it was possible for data to
remain in internal buffers. This data would be emitted on a subsequent call
to decompress(). The overall output stream would still be valid. But if
callers were expecting input data to exactly map to output data (say the
producer had used flush(COMPRESSOBJ_FLUSH_BLOCK) and was attempting to
map input chunks to output chunks), then the previous behavior would be
wrong. The new behavior is such that output from
flush(COMPRESSOBJ_FLUSH_BLOCK) fed into decompressobj().decompress()
should produce all available compressed input.
* ZstdDecompressor.stream_reader().read() should no longer segfault after
a previous context manager resulted in error.
* ZstdCompressor.compressobj().flush(COMPRESSOBJ_FLUSH_BLOCK) now returns
all data necessary to flush a block. Before, it was possible for the
flush() to not emit all data necessary to fully represent a block. This
would mean decompressors wouldn't be able to decompress all data that had been
fed into the compressor and flush()ed.
New Features
* New module constants BLOCKSIZELOG_MAX, BLOCKSIZE_MAX,
TARGETLENGTH_MAX that expose constants from libzstd.
* New ZstdCompressor.chunker() API for manually feeding data into a
compressor and emitting chunks of a fixed size. Like compressobj(), the
API doesn't impose restrictions on the input or output types for the
data streams. Unlike compressobj(), it ensures output chunks are of a
fixed size. This makes this API useful when the compressed output is being
fed into an I/O layer, where uniform write sizes are useful.
* ZstdCompressor.stream_reader() no longer needs to be used as a context
manager.
* ZstdDecompressor.stream_reader() no longer needs to be used as a context
manager.
* Bundled zstandard library upgraded from 1.3.4 to 1.3.6.
Changes
* Added zstd_cffi.py and NEWS.rst to MANIFEST.in.
* zstandard.__version__ is now defined.
* Upgrade pip, setuptools, wheel, and cibuildwheel packages to latest versions.
* Upgrade various packages used in CI to latest versions. Notably tox (in
order to support Python 3.7).
* Use relative paths in setup.py to appease Python 3.7.
* Added CI for Python 3.7.
Zstandard v1.3.7
perf: slightly better decompression speed on clang (depending on hardware target)
fix: ratio for dictionary compression at levels 9 and 10, reported by @indygreg
build: no longer build backtrace by default in release mode; restrict further automatic mode
build: control backtrace support through build macro BACKTRACE
misc: added man pages for zstdless and zstdgrep, by @samrussell
Zstandard v1.3.6 release is focused on intensive dictionary compression for database scenarios.
This is a new environment we are experimenting. The success of dictionary compression on small data, of which databases tend to store plentiful, led to increased adoption, and we now see scenarios where literally thousands of dictionaries are being used simultaneously, with permanent generation or update of new dictionaries.
== 1.0.0 (2018-05-20)
* *BreakingChange* The XZ module's methods now take any parameters
beyond the IO object as real Ruby keyword arguments rather than
a long argument list.
* *BreakingChange* XZ.decompress_stream now honours Ruby's
external and internal encoding concept instead of just
returning BINARY-tagged strings.
* *BreakingChange* Remove deprecated API on stream reader/writer
class and instead sync the API with Ruby's zlib library
(Ticket #12 by me).
* *BreakingChange* StreamWriter.new and StreamReader.new do not accept
a block anymore. This is part of syncing with Ruby's zlib API.
* *BreakingChange* StreamReader.open and StreamWriter.open always
return the new instance, even if a block is given to the method
(previous behaviour was to return the return value of the block).
This is part of the syncing with Ruby's zlib API.
* *BreakingChange* StreamReader.new and StreamWriter.new as well as
the ::open variants take additional arguments as real Ruby keyword
arguments now instead of a long parameter list plus options hash.
This is different from Ruby's own zlib API as that one takes both
a long parameter list and a hash of additional options. ruby-xz
is meant to follow zlib's semantics mostly, but not as a drop-in
replacement, so this divergence from zlib's API is okay (also
given that it isn't possible to replicate all possible options
1:1 anyway, since liblzma simply accepts different options as
libz). If you've never used these methods' optional arguments,
you should be fine.
* *BreakingChange* Stream#close now returns nil instead of the
number of bytes written. This syncs Stream#close with Ruby's
own IO#close, which also returns nil.
* *BreakingChange* Remove Stream#pos=, Stream#seek, Stream#stat. These
methods irritated the minitar gem, which doesn't expect them to
raise NotImplementedError, but directly to be missing if the object
does not support seeking.
* *BreakingChange* StreamReader and StreamWriter now honour Ruby's
encoding system instead of returning only BINARY-tagged strings.
* *Dependency* Remove dependency on ffi. ruby-xz now uses fiddle from
the stdlib instead.
* *Dependency* Remove dependency on io-like. ruby-xz now implements
all the IO mechanics itself. (Ticket #10 by me)
* *Dependency* Bump required Ruby version to 2.3.0.
* *Fix* libzlma.dylib not being found on OS X (Ticket #15 by
s0nspark).
- perf: minor decompression speed improvement (~+2%) with gcc
- fix : corruption in v1.8.2 at level 9 for files > 64KB under rare
conditions (#560)
- cli : new command --fast, by @jennifermliu
- api : LZ4_decompress_safe_partial() now decodes exactly the nb of
bytes requested (feature request #566)
- build : added Haiku target, by @fbrosson, and MidnightBSD, by @laffer1
- doc : updated documentation regarding dictionary compression
This is based on the decision The NetBSD Foundation made in 2008 to
do so, which was already applied to src.
This change has been applied to code which is likely not in other
repositories.
ok board@, reviewed by riastradh@
1.61 Sat 18 Aug 2018
- File::Find will not untaint [github/ThisUsedToBeAnEmail]
- Prevent from traversing symlinks and parent directories when extracting [github/ppisar]
Changes:
improve q=1 compression on small files
inverse Bazel workspace tree
add rolling-composite-hasher for large-window mode
add tools to download and transform static dictionary data
Changes:
2018-03-15 guidod <guidod@gmx.de>
* fix a number of CVEs reported with special *.zip PoC files
* man-pages are generated with new dbk2man.py - docbook xmlto is optional now
* completing some doc strings while checking the new man-pages to look good
* allow the zziptests.py testsuite to run with an installed /bin path
* try to fix some issues on testing with non-installed binaries on non-linux platfors
* update autotools to allow compiling on some newer Mac / Win machines
* a zip-program is still required for testing, but some errors are gone when not there
* complete the approximation of fnmatch for the test binaries (on platforms without)
* allow windows __mmap.h to be simpler, helping with some problems on MingW
* integrate 'fopen("wb")' from TexLive to be more portable across
* more portability as well for helpers like strnlen being used in the sources
* update doc refs to point to github instead of sf.net
* update the sf.net pages to have a prominent hint on newer github.com location
* release v0.13.69
2018-04-26 Stuart Caie <kyzer@cabextract.org.uk>
* read_chunk(): the test that chunk numbers are in bounds was off
by one, so read_chunk() returned a pointer taken from outside
allocated memory that usually crashes libmspack when accessed.
Thanks to Hanno Böck for finding the issue and providing a sample.
* chmd_read_headers(): reject files with blank filenames. Thanks
again to Hanno Böck for finding the issue and providing a sample file.
2018-02-06 Stuart Caie <kyzer@cabextract.org.uk>
* chmd.c: fixed an off-by-one error in the TOLOWER() macro, reported
by Dmitry Glavatskikh. Thanks Dmitry!
2017-11-26 Stuart Caie <kyzer@cabextract.org.uk>
* kwajd_read_headers(): fix up the logic of reading the filename and
extension headers to avoid a one or two byte overwrite. Thanks to
Jakub Wilk for finding the issue.
* test/kwajd_test.c: add tests for KWAJ filename.ext handling
2017-10-16 Stuart Caie <kyzer@cabextract.org.uk>
* test/cabd_test.c: update the short string tests to expect not only
MSPACK_ERR_DATAFORMAT but also MSPACK_ERR_READ, because of the recent
change to cabd_read_string(). Thanks to maitreyee43 for spotting this.
* test/msdecompile_md5: update the setup instructions for this script,
and also change the script so it works with current Wine. Again, thanks
to maitreyee43 for trying to use it and finding it not working.
2017-08-13 Stuart Caie <kyzer@cabextract.org.uk>
* src/chmextract.c: support MinGW one-arg mkdir(). Thanks to AntumDeluge
for reporting this.
2017-08-13 Stuart Caie <kyzer@cabextract.org.uk>
* read_spaninfo(): a CHM file can have no ResetTable and have a
negative length in SpanInfo, which then feeds a negative output length
to lzxd_init(), which then sets frame_size to a value of your choosing,
the lower 32 bits of output length, larger than LZX_FRAME_SIZE. If the
first LZX block is uncompressed, this writes data beyond the end of the
window. This issue was raised by ClamAV as CVE-2017-6419. Thanks to
Sebastian Andrzej Siewior for finding this by chance!
* lzxd_init(), lzxd_set_output_length(), mszipd_init(): due to the issue
mentioned above, these functions now reject negative lengths
2017-08-05 Stuart Caie <kyzer@cabextract.org.uk>
* cabd_read_string(): add missing error check on result of read().
If an mspack_system implementation returns an error, it's interpreted
as a huge positive integer, which leads to reading past the end of the
stack-based buffer. Thanks to Sebastian Andrzej Siewior for explaining
the problem. This issue was raised by ClamAV as CVE-2017-11423
2016-04-20 Stuart Caie <kyzer@cabextract.org.uk>
* configure.ac: change my email address to kyzer@cabextract.org.uk
2015-05-10 Stuart Caie <kyzer@4u.net>
* cabd_read_string(): correct rejection of empty strings. Thanks to
Hanno Böck for finding the issue and providing a sample file.
2015-05-10 Stuart Caie <kyzer@4u.net>
* Makefile.am: Add subdir-objects option as suggested by autoreconf.
* configure.ac: Add AM_PROG_AR as suggested by autoreconf.
2015-01-29 Stuart Caie <kyzer@4u.net>
* system.h: if C99 inttypes.h exists, use its PRI{d,u}{32,64} macros.
Thanks to Johnathan Kollasch for the suggestion.
New in 1.7
* cabextract now supports an --encoding parameter, to specify the character
encoding of CAB filenames if they are not ASCII or UTF8
* cabextract -L now lowercases non-ASCII characters
Performing substitutions during post-patch breaks tools such as mkpatches,
making it very difficult to regenerate correct patches after making changes,
and often leading to substituted string replacements being committed.
Update LICENSE
Upstream changes:
0.26 (2018/06/09)
Implemented refactoring due warnings from Perl::Critic.
0.25 (2018/06/04)
Implemented refactoring due warnings from Perl::Critic.
Merge pull request #3 from manwar/suggest-code-tidy
0.24 (2018/06/02)
Added a LICENSE file (GNU GPL v3).
Removed MYMETA files (see https://rt.cpan.org/Ticket/Display.html?id=108171).
Improved Kwalitee by adding information to Makefile.PL
Fixed tests under OpenBSD
Added some code to check for OpenBSD tar, which is not quite compatible to the command line options passed by this module.
Also made the method is_gnu() more robust, testing the return code and properly handling STDOUT and STDERR when trying "tar --version".
Dependencies added are those already available on standard perl (Config and IPC::Open3).
Added a README.md for better formatting in Github project page.
Small refactorings and code formating with perltidy.
Upstream changes:
2.30 19/06/2018
- skip white_space test on MSWin32 as Windows will report that both
files exist, which is obviously a 'feature'
2.28 08/06/2018 (madroach, ARC, OCBNET, ppisar)
- fix creating file with trailing whitespace on filename - fixes 103279
- allow archiving with absolute pathnames - fixes 97748
- small POD fix
- Speed up extract when archive contains lots of files
- CVE-2018-12015 directory traversal vulnerability [RT#125523]
2.0.1:
This release fixes: tests failed when run under python setup.py test, but passed when running under tox.
2.0.0:
It's now possible to specify a compession dictionary for block compression.
The bundled LZ4 libraries have been updated to 1.8.2
A compatibility fix for 2.x memoryview objects has been added.
Various flake8 cleanups and test additions.
This Go language package supports the reading and writing of xz
compressed streams. It includes also a gxz command for compressing and
decompressing data. The package is completely written in Go and
doesn't have any dependency on any C code.
Changes 2.8:
add support for setting atime, ctime, mtime and birthtime
tell libarchive when writing an archive is aborted due to an exception
add support for getting uid and gid
add support for high resolution timestamps
add two new archive readers: stream_reader and custom_reader
add missing archive extraction flags
add the lz4 and warc formats
add support for write options and uid/gid lookup
innoextract 1.7 (2018-06-12)
- Added support for Inno Setup 5.6.0 installers
- Added support for new GOG installers with GOG Galaxy file parts
- Added support for encrypted installers with the --password (-P) and --password-file options
- Added a --show-password option to print password check information
- Added a --check-password option to abort if the provided password does not match the stored checksum
- Added a --info (-i) convenience option to print information about the installer
- Added a --list-sizes option to print file sizes even with --quiet or --silent
- Added a --list-checksums option to print file checksums
- Added a --data-version (-V) option to print the data version and exit
- Added a --no-extract-unknown (-n) option to abort on unknown Inno Setup data versions
- Fixed building in paths that contain regex expressions
- Fixed case-sensitivity in parent directory when creating subdirectories
- Fixed .bin slice file names used with Inno Setup versions older than 4.1.7
- Fixed build with newer libc++ versions
- Made loading of .bin slice files case-insensitive
- The --test option can now be combined with --extract to abort on file checksum errors
- Now compiles in C++17 mode if supported
5.2.4:
* liblzma:
- Allow 0 as memory usage limit instead of returning
LZMA_PROG_ERROR. Now 0 is treated as if 1 byte was specified,
which effectively is the same as 0.
- Use "noexcept" keyword instead of "throw()" in the public
headers when a C++11 (or newer standard) compiler is used.
- Added a portability fix for recent Intel C Compilers.
- Microsoft Visual Studio build files have been moved under
windows/vs2013 and windows/vs2017.
* xz:
- Fix "xz --list --robot missing_or_bad_file.xz" which would
try to print an unitialized string and thus produce garbage
output. Since the exit status is non-zero, most uses of such
a command won't try to interpret the garbage output.
- "xz --list foo.xz" could print "Internal error (bug)" in a
corner case where a specific memory usage limit had been set.
Engrampa, the archive viewer, has improved support for encrypted 7z archives.
Full changelog:
build: use PKG_CONFIG to fix cross-build
Add our copyright to About dialog and Caja extension
7z: Fix: rename files with password without the list encrypted
7z: Fix: delete/rename files/folders with the list encrypted
avoid deprecated gdk_screen_make_display_name
don’t use deprecated gtk_show_uri
use a more common gtk+ function
avoid deprecated gdk_screen_get_number
Add the button “Show the Files and Quit” in the progress dialog
Fix: create zip files in “maximum” compression level
Fix: Browsing history not correct
hide folders in “View All Files”
Fix: Wrong behavior of Skip button in Replace file dialog
UI files: avoid deprecations
gtk-utils: remove some GTK_STOCK deprecations
gtk-utils: avoid deprecated gtk_icon_size_lookup_for_settings
fr-window: fix some GTK_STOCK deprecations
add style class frame to scrolledwindows
fr-window: avoid deprecated GtkMisc and GtkAlignment
dlg-add-folder: avoid deprecated gtk_alignment_new()
build: use variable instead of hardcoded file name when cleaning
Translations update
v1.8.2
perf: *much* faster dictionary compression on small files
perf: improved decompression speed and binary size
perf: slightly faster HC compression and decompression speed
perf: very small compression ratio improvement
fix : compression compatible with low memory addresses (< 0xFFFF)
fix : decompression segfault when provided with NULL input
cli : new command --favor-decSpeed
cli : benchmark mode more accurate for small inputs
fullbench : can bench _destSize() variants
doc : clarified block format parsing restrictions
1.5.1 [2018-04-11]
==================
* Choose format of installed documentation based on available tools.
* Fix visibility of symbols.
* Fix zipcmp directory support.
* Don't set RPATH on Linux.
* Use Libs.private for link dependencies in pkg-config file.
* Fix build with LibreSSL.
* Various bugfixes.
0.9.0:
Backwards Compatibility Notes
CFFI 1.11 or newer is now required (previous requirement was 1.8).
The primary module is now zstandard. Please change imports of zstd and zstd_cffi to import zstandard. See the README for more. Support for importing the old names will be dropped in the next release.
ZstdCompressor.read_from() and ZstdDecompressor.read_from() have been renamed to read_to_iter(). read_from() is aliased to the new name and will be deleted in a future release.
Support for Python 2.6 has been removed.
Support for Python 3.3 has been removed.
The selectivity argument to train_dictionary() has been removed, as the feature disappeared from zstd 1.3.
Support for legacy dictionaries has been removed. Cover dictionaries are now the default. train_cover_dictionary() has effectively been renamed to train_dictionary().
The allow_empty argument from ZstdCompressor.compress() has been deleted and the method now allows empty inputs to be compressed by default.
estimate_compression_context_size() has been removed. Use CompressionParameters.estimated_compression_context_size() instead.
get_compression_parameters() has been removed. Use CompressionParameters.from_level() instead.
The arguments to CompressionParameters.__init__() have changed. If you were using positional arguments before, the positions now map to different arguments. It is recommended to use keyword arguments to construct CompressionParameters instances.
TARGETLENGTH_MAX constant has been removed (it disappeared from zstandard 1.3.4).
ZstdCompressor.write_to() and ZstdDecompressor.write_to() have been renamed to ZstdCompressor.stream_writer() and ZstdDecompressor.stream_writer(), respectively. The old names are still aliased, but will be removed in the next major release.
Content sizes are written into frame headers by default (ZstdCompressor(write_content_size=True) is now the default).
CompressionParameters has been renamed to ZstdCompressionParameters for consistency with other types. The old name is an alias and will be removed in the next major release.
Bug Fixes
Fixed memory leak in ZstdCompressor.copy_stream().
Fixed memory leak in ZstdDecompressor.copy_stream().
Fixed memory leak of ZSTD_DDict instances in CFFI's ZstdDecompressor.
New Features
Bundlded zstandard library upgraded from 1.1.3 to 1.3.4. This delivers various bug fixes and performance improvements. It also gives us access to newer features.
Support for negative compression levels.
Support for long distance matching (facilitates compression ratios that approach LZMA).
Supporting for reading empty zstandard frames (with an embedded content size of 0).
Support for writing and partial support for reading zstandard frames without a magic header.
New stream_reader() API that exposes the io.RawIOBase interface (allows you to .read() from a file-like object).
Several minor features, bug fixes, and performance enhancements.
Wheels for Linux and macOS are now provided with releases.
Changes
Functions accepting bytes data now use the buffer protocol and can accept more types (like memoryview and bytearray).
Add #includes so compilation on OS X and BSDs works.
New ZstdDecompressor.stream_reader() API to obtain a read-only i/o stream of decompressed data for a source.
New ZstdCompressor.stream_reader() API to obtain a read-only i/o stream of compressed data for a source.
Renamed ZstdDecompressor.read_from() to ZstdDecompressor.read_to_iter(). The old name is still available.
Renamed ZstdCompressor.read_from() to ZstdCompressor.read_to_iter(). read_from() is still available at its old location.
Introduce the zstandard module to import and re-export the C or CFFI backend as appropriate. Behavior can be controlled via the PYTHON_ZSTANDARD_IMPORT_POLICY environment variable. See README for usage info.
Vendored version of zstd upgraded to 1.3.4.
Added module constants CONTENTSIZE_UNKNOWN and CONTENTSIZE_ERROR.
Add STRATEGY_BTULTRA compression strategy constant.
Switch from deprecated ZSTD_getDecompressedSize() to ZSTD_getFrameContentSize() replacement.
ZstdCompressor.compress() can now compress empty inputs without requiring special handling.
ZstdCompressor and ZstdDecompressor now have a memory_size() method for determining the current memory utilization of the underlying zstd primitive.
train_dictionary() has new arguments and functionality for trying multiple variations of COVER parameters and selecting the best one.
Added module constants LDM_MINMATCH_MIN, LDM_MINMATCH_MAX, and LDM_BUCKETSIZELOG_MAX.
Converted all consumers to the zstandard new advanced API, which uses ZSTD_compress_generic()
CompressionParameters.__init__ now accepts several more arguments, including support for long distance matching.
ZstdCompressionDict.__init__ now accepts a dict_type argument that controls how the dictionary should be interpreted. This can be used to force the use of content-only dictionaries or to require the presence of the dictionary magic header.
ZstdCompressionDict.precompute_compress() can be used to precompute the compression dictionary so it can efficiently be used with multiple ZstdCompressor instances.
Digested dictionaries are now stored in ZstdCompressionDict instances, created automatically on first use, and automatically reused by all ZstdDecompressor instances bound to that dictionary.
All meaningful functions now accept keyword arguments.
ZstdDecompressor.decompressobj() now accepts a write_size argument to control how much work to perform on every decompressor invocation.
ZstdCompressor.write_to() now exposes a tell(), which exposes the total number of bytes written so far.
ZstdDecompressor.stream_reader() now supports seek() when moving forward in the stream.
Removed TARGETLENGTH_MAX constant.
Added frame_header_size(data) function.
Added frame_content_size(data) function.
Consumers of ZSTD_decompress* have been switched to the new advanced decompression API.
ZstdCompressor and ZstdCompressionParams can now be constructed with negative compression levels.
ZstdDecompressor now accepts a max_window_size argument to limit the amount of memory required for decompression operations.
FORMAT_ZSTD1 and FORMAT_ZSTD1_MAGICLESS constants to be used with the format compression parameter to control whether the frame magic header is written.
ZstdDecompressor now accepts a format argument to control the expected frame format.
ZstdCompressor now has a frame_progression() method to return information about the current compression operation.
Error messages in CFFI no longer have b'' literals.
Compiler warnings and underlying overflow issues on 32-bit platforms have been fixed.
Builds in CI now build with compiler warnings as errors. This should hopefully fix new compiler warnings from being introduced.
Make ZstdCompressor(write_content_size=True) and CompressionParameters(write_content_size=True) the default.
CompressionParameters has been renamed to ZstdCompressionParameters.
1.1.0:
This release removes the deprecated functions which were marked as remove in 1.0, but nonetheless remained:
lz4.lz4version()
LZ4FrameCompressor.finalize()
As a side effect, we noo longer have a dependency on the deprecation package.
1.5.0 [2018-03-11]
==================
* Use standard cryptographic library instead of custom AES implementation.
This also simplifies the license.
* Use `clang-format` to format the source code.
* More Windows improvements.
version 1.30 - Sergey Poznyakoff, 2017-12-17
* Member names containing '..' components are now skipped when extracting.
This fixes tar's behavior to match its documentation, and is a bit
safer when extracting untrusted archives over old files (an unsafe
practice that the tar manual has long recommended against).
* Report erroneous use of position-sensitive options.
During archive creation or update, tar keeps track of positional
options (see the manual, subsection 3.4.4 "Position-Sensitive
Options"), and reports those that had no effect. For example, when
invoked as
tar -cf a.tar . --exclude '*.o'
tar will create the archive, but will exit with status 2, having
issued the following error message
tar: The following options were used after non-optional
arguments in archive create or update mode. These options are
positional and affect only arguments that follow them. Please,
rearrange them properly.
tar: --exclude '*.o' has no effect
tar: Exiting with failure status due to previous errors
* --numeric-owner now affects private headers too.
This helps the output of 'tar' to be more deterministic.
* Fixed the --delay-directory-restore option
In some cases tar would restore the directory permissions too early,
causing subsequent link extractions in that directory to fail.
* The --warnings=failed-read option
This new warning control option suppresses warning messages about
unreadable files and directories. It has effect only if used together
with the --ignore-failed-read option.
* The --warnings=none option now suppresses all warnings
This includes warnings about unreadable files produced when
--ignore-failed-read is in effect. To output these, use
--warnings=none --warnings=no-failed-read.
* Fix reporting of hardlink mismatches during compare
Tar reported incorrect target file name in the 'Not linked to'
diagnostic message.
Changes in version 1.20:
The option '--loose-trailing', has been added.
The test used by lzip to discriminate trailing data from a corrupt
header in multimember or concatenated files has been improved to a
Hamming distance (HD) of 3, and the 3 bit flips must happen in different
magic bytes for the test to fail. As a consequence some kinds of files
no longer can be appended to a lzip file as trailing data unless the
'--loose-trailing' option is used when decompressing.
Lziprecover can be used to remove conflicting trailing data from a file.
The contents of a corrupt or truncated header found in a multimember
file are now shown, after the error message, in the same format as
trailing data.
Option '-S, --volume-size' now keeps input files unchanged.
When creating multimember files or splitting the output in volumes, the
dictionary size is now adjusted for each member individually.
The 'bits/byte' ratio has been replaced with the inverse compression
ratio in the output.
The progress of decompression is now shown at verbosity level 2 (-vv) or
higher.
Progress of (de)compression is only shown if stderr is a terminal.
A final diagnostic is now shown at verbosity level 1 (-v) or higher if
any file fails the test when testing multiple files.
A second '.lz' extension is no longer added to the argument of '-o' if
it already ends in '.lz' or '.tlz'.
In case of (de)compressed size mismatch, the stored size is now also
shown in hexadecimal to ease visual comparison.
The dictionary size is now shown at verbosity level 4 (-vvvv) when
decompressing or testing.
The new chapter "Meaning of lzip's output" has been added to the manual.
Changelog:
2018-02-02 guidod <guidod@gmx.de>
* fix a number of CVEs reported with special *.zip files
* the testsuite has been expanded to cover all the CVEs
* some minor doc updates referencing GitHub instead of sf.net
* release v0.13.68
0.23.2:
Fixes an error in the deprecated LZ4Compressor.finalize() method
Improves documentation
Has all example code in documentation verified via doctest
Skip running the regression tests since for some reason the setting
of LD_LIBRARY_PATH isn't passed down through cmake invocation; this
avoids error:
dyld: Library not loaded: @rpath/libzip.5.dylib
Referenced from: ${WRKSRC}/regress/../src/ziptool
Reason: image not found
The module has bundled version 1.1.2, but the configure script warns
when using the bundled one, and it's of course better to not duplicate
it. Noticed this on recent PHP 7.1.14/7.2.2 releases which had fixes
for zip extension when used with libzip >= 1.3.1.
XXX May need backport of the fix for 7.0 and 5.6 which did not get the fix,
or can just switch over to using the PECL module directly
0.19.1:
This release adds compressed file handling capability to the lz4.frame sub-package.
This necessitated some changes to the API of lz4.frame.decompress_chunk, ad some smaller changes to the LZ4FrameCompressor and LZ4FrameDecompressor classes. Please see updated documentation for further details.
This is a build-maintenance release. Major changes:
added Autotools build files
switched shared library version to libtool scheme
In this release semantic suffix and libtool suffix are the same: 1.0.2.
Don't expect them to match in future releases.
Minor changes:
BrotliDictionary members are not const now
ZopfliNode distance could be up to 128MiB
fixed API documentation typos
total_out is always set by decoder
fixed BROTLI_ENSURE_CAPACITY macro; no-op in preprocessed output
Other changes:
fixed scripts for oss-fuzz, test them with Travis
made Bazel JNI tests less messy
fixed linter warnings in JS decoder
fixed permissions of various files
added Bazel build to Appveyor matrix
added Sieve dictionary generator
0.18.2:
This release fixes a memory leak that was introduced in lz4.frame.decompress in 0.18.1. This leak resulted from an incorrect ref count on the returned result which prevented it from ever being released and garbage collected.
v1.8.1.2:
It's the same as v1.8.1, but the version number in source code has been fixed.
The version number is used in cli and documentation display, to create the full name of dynamic library, and can be requested via LZ4_versionNumber().
v1.8.1
perf : faster and stronger ultra modes (levels 10+)
perf : slightly faster compression and decompression speed
perf : fix bad degenerative case
fix : decompression failed when using a combination of extDict + low memory address
cli : support for dictionary compression (-D)
cli : fix : lz4 -d --rm preserves timestamp
cli : fix : do not modify /dev/null permission as root
api : _destSize() variant supported for all compression levels
build : make and make test compatible with -jX
build : can control LZ4LIB_VISIBILITY macro
install: fix man page directory
The actual fix as been done by "pkglint -F */*/buildlink3.mk", and was
reviewed manually.
There are some .include lines that still are indented with zero spaces
although the surrounding .if is indented. This is existing practice.
- Generate Zip64 format .zip files when the output is greater than
or equal to 4 GiB.
- Improved gzip options processing and return code compatibility
- Some bug fixes
0.18.1:
This is a minor bugfix release which fixes two small bugs which would result in source references not being released on error conditions in lz4.frame.compress.
0.18.0:
This release changes the strategy for allocating memory to not use undocumented and potentially fragile Python internals (Py_SIZE). This may have a small performance impact, but brings stronger guarantees for future stability.
Unsorted entries in PLIST files have generated a pkglint warning for at
least 12 years. Somewhat more recently, pkglint has learned to sort
PLIST files automatically. Since pkglint 5.4.23, the sorting is only
done in obvious, simple cases. These have been applied by running:
pkglint -Cnone,PLIST -Wnone,plist-sort -r -F
0.17.0:
This release fixes a pathological case where compressing and decompressing a zero length bytes object could result in the resulting bytes object causing False to be returned from val == 'b'.
1.4.0 [2017-12-29]
==================
* Improve build with cmake
* Retire autoconf/automake build system
* Add `zip_source_buffer_fragment()`.
* Add support to clone unchanged beginning of archive (instead of rewriting it).
Supported for buffer sources and on Apple File System.
* Add support for Microsoft Universal Windows Platform.
1.3.3:
perf: improved zstd_opt strategy (levels 16-19)
fix : bug 944 : multithreading with shared ditionary and large data
cli : fix : content size written in header by default
cli : fix : improved LZ4 format support
cli : new : hidden command -b -S, to benchmark multiple files and generate one result per file
api : change : when setting pledgedSrcSize, use ZSTD_CONTENTSIZE_UNKNOWN macro value to mean "unknown"
api : fix : support large skippable frames
api : fix : re-using context could result in suboptimal block size in some corner case scenarios
api : fix : streaming interface was adding a useless 3-bytes null block to small frames
build: fix : compilation under rhel6 and centos6
build: added check target
build: improved meson support
This release brings
Huge changes to the LZ4 Frame support which should now be considered beta quality
A new pytest based test harness, and significantly re-worked testing
An end to Python 2.6 support - no testing is currently done against Python 2.6 although the code probably does still work with Python 2.6 at this point in time.
atool is a script for managing file archives of various types (tar,
tar+gzip, zip, etc.). It provides the following commands:
* aunpack: extract an archive, avoiding tar bombs
* apack: create archives or compressed files
* als: list the contents of an archive
* acat: extract to the standard output
* adiff: generate a diff between two archives
* arepack: repack archives to a different format
Optional dependencies: lbzip2 or pbzip2, lzip, plzip, lzop, lzma, zip,
unzip, unrar, lha, unace, arj, arc, nomarch, p7zip, unalz.
Brotli is a generic-purpose lossless compression algorithm that compresses data
using a combination of a modern variant of the LZ77 algorithm, Huffman coding
and 2nd order context modeling, with a compression ratio comparable to the best
currently available general-purpose compression methods. It is similar in speed
with deflate but offers more dense compression.
0.8.1:
Add #includes so compilation on OS X and BSDs works.
0.8.0:
CompressionParameters now has a estimated_compression_context_size() method. zstd.estimate_compression_context_size() is now deprecated and slated for removal.
Implemented a lot of fuzzing tests.
CompressionParameters instances now perform extra validation by calling ZSTD_checkCParams() at construction time.
multi_compress_to_buffer() API for compressing multiple inputs as a single operation, as efficiently as possible.
ZSTD_CStream instances are now used across multiple operations on ZstdCompressor instances, resulting in much better performance for APIs that do streaming.
ZSTD_DStream instances are now used across multiple operations on ZstdDecompressor instances, resulting in much better performance for APIs that do streaming.
train_dictionary() now releases the GIL.
Support for training dictionaries using the COVER algorithm.
multi_decompress_to_buffer() API for decompressing multiple frames as a single operation, as efficiently as possible.
Support for multi-threaded compression.
Disable deprecation warnings when compiling CFFI module.
Fixed memory leak in train_dictionary().
Removed DictParameters type.
train_dictionary() now accepts keyword arguments instead of a DictParameters instance to control dictionary generation.
v1.3.2
new : long range mode, using --long command
new : ability to generate and decode magicless frames
changed : maximum nb of threads reduced to 200, to avoid address space exhaustion in 32-bits mode
fix : multi-threading compression works with custom allocators
fix : ZSTD_sizeof_CStream() was over-evaluating memory usage
fix : a rare compression bug when compression generates very large distances and bunch of other conditions (only possible at --ultra -22)
fix : 32-bits build can now decode large offsets (levels 21+)
cli : added LZ4 frame support by default
cli : improved --list output
cli : new : can split input file for dictionary training, using command -B#
cli : new : clean operation artefact on Ctrl-C interruption
cli : fix : do not change /dev/null permissions when using command -t with root access
cli : fix : write file size in header in multiple-files mode
api : added macro ZSTD_COMPRESSBOUND() for static allocation
api : experimental : new advanced decompression API
api : fix : sizeof_CCtx() used to over-estimate
build: fix : no-multithread variant compiles without pool.c dependency
build: better compatibility with reproducible builds
example : added streaming_memory_usage
license : changed /examples license to BSD + GPLv2
license : fix a few header files to reflect new license
py-lz4 provides Python bindings for the LZ4 compression library by Yann
Collet.
At this time the project contains bindings for the LZ4 block format,
which are considered stable.
Support for the LZ4 frame format is a work-in-progress and available as a
technology preview. A future relase will implement support for the LZ4
stream format.
Temporarily remove the zstd buildlink as py-zstandard includes
its own version that it wants to use. Future versions will
support using a system supplied zstd but this one doesn't yet.
Also prevent setup.py from picking up cffi by accident so the
PLIST is stable.
