Zstandard v1.3.8
perf: better decompression speed on large files (+7%) and cold dictionaries (+15%)
perf: slightly better compression ratio at high compression modes
api : finalized advanced API, last stage before "stable" status
api : new --rsyncable mode
api : support decompression of empty frames into NULL (used to be an error)
build: new set of build macros to generate a minimal size decoder
build: fix compilation on MIPS32
build: fix compilation with multiple -arch flags
build: highly upgraded meson build
build: improved buck support
build: fix cmake script : can create debug build
build: Makefile : grep works on both colored consoles and systems without color support
build: fixed zstd-pgo target
cli : support ZSTD_CLEVEL environment variable
cli : --no-progress flag, preserving final summary
cli : ensure destination file is not source file
cli : clearer error messages, notably when input file not present
doc : clarified zstd_compression_format.md
misc: fixed zstdgrep, returns 1 on failure
misc: NEWS renamed as CHANGELOG, in accordance with fb.oss policy
2.1.5
This release contains no functional changes other than changes to the Appveyor configuration for publishing wheels.
2.1.4
This release contains no functional changes other than changes to the Travis configuration for publishing wheels.
2.1.3
A simplification of the tox.ini file
More robust checking for pkgconfig availability
Integration of cibuildwheel into travis builds so as to build and publish binary wheels for Linux and OSX
Only require pytest-runner if pytest/test is being called
Blacklists version 3.3.0 of pytest which has a bug that can cause the tests to fail.
1.0.7
cross compilation support:
added ability to run cross-compiled ARM tests in qemu
added arm-linux-gnueabihf-gcc entry to Travis build matrix
faster decoding on ARM:
implemented prefetching HuffmanCode entry as uint32_t if target platform is ARM
fixed NEON extension detection
combed Huffman table building code for better readability
improved precision of window size calculation in CLI
minor fixes:
fixed typos
improved internal comments / parameter names
fixed BROTLI_PREDICT_TRUE/_FALSE detection for SunPro compiler
unburdened JNI (Bazel) builds from fetching the full JDK
1.0.6
Fixes
fix unaligned 64-bit accesses on AArch32
add missing files to the sources list
add ASAN/MSAN unaligned read specializations
fix CoverityScan "unused assignment" warning
fix JDK 8<->9 incompatibility
unbreak Travis builds
fix auto detect of bundled mode in cmake
* libmspack is now distributed with its test-suite, which now run
as part of "make check"
* libmspack's programs in src/ have been moved to examples/ and do
not auto-install
Set TEST_TARGET.
New in 1.9
* Fixed invisible bad extraction when using cabextract -F (broken in 1.8)
* Fixed configure --with-external-libmspack which was broken in 1.8
* configure --with-external-libmspack will now use pkg-config. To configure
it manually, set environment variables libmspack_CFLAGS and libmspack_LIBS
before running configure.
* Now includes the test suite (make check)
New in 1.8
* cabextract -f now extracts even more badly damaged files than before
Uptsream changes:
2.32 13/09/2018 (CBERRY)
- Fix absolute path handling on VMS
2.30 19/06/2018
- skip white_space test on MSWin32 as Windows will report that both
files exist, which is obviously a 'feature'
2.1.2:
Improves the speed of importing the module by avoiding the use of pkg_resources
Fixes some flake8 warnings
Resolves a small issue with the test suite when detecting memory usage increases
* On Arch Linux, the build failed, makedev(3) indicates
#include <sys/sysmacros.h>
* On Debian Buster, the build succeed but a big warning is displayed:
warning: In the GNU C Library, "minor" is defined
by <sys/sysmacros.h>. For historical compatibility, it is
currently defined by <sys/types.h> as well, but we plan to
remove this soon. To use "minor", include <sys/sysmacros.h>
directly. If you did not intend to use a system-defined macro
"minor", you should undefine it after including <sys/types.h>.
0.10.1:
Backwards Compatibility Notes
* ZstdCompressor.stream_reader().closed is now a property instead of a
method.
* ZstdDecompressor.stream_reader().closed is now a property instead of a
method.
Changes
* Stop attempting to package Python 3.6 for Miniconda. The latest version of
Miniconda is using Python 3.7. The Python 3.6 Miniconda packages were a lie
since this were built against Python 3.7.
* ZstdCompressor.stream_reader()'s and ZstdDecompressor.stream_reader()'s
closed attribute is now a read-only property instead of a method. This now
properly matches the IOBase API and allows instances to be used in more
places that accept IOBase instances.
0.10.0:
Backwards Compatibility Notes
* ZstdDecompressor.stream_reader().read() now consistently requires an
argument in both the C and CFFI backends. Before, the CFFI implementation
would assume a default value of -1, which was later rejected.
* The compress_literals argument and attribute has been removed from
zstd.ZstdCompressionParameters because it was removed by the zstd 1.3.5
API.
* ZSTD_CCtx_setParametersUsingCCtxParams() is no longer called on every
operation performed against ZstdCompressor instances. The reason for this
change is that the zstd 1.3.5 API no longer allows this without calling
ZSTD_CCtx_resetParameters() first. But if we called
ZSTD_CCtx_resetParameters() on every operation, we'd have to redo
potentially expensive setup when using dictionaries. We now call
ZSTD_CCtx_reset() on every operation and don't attempt to change
compression parameters.
* Objects returned by ZstdCompressor.stream_reader() no longer need to be
used as a context manager. The context manager interface still exists and its
behavior is unchanged.
* Objects returned by ZstdDecompressor.stream_reader() no longer need to be
used as a context manager. The context manager interface still exists and its
behavior is unchanged.
Bug Fixes
* ZstdDecompressor.decompressobj().decompress() should now return all data
from internal buffers in more scenarios. Before, it was possible for data to
remain in internal buffers. This data would be emitted on a subsequent call
to decompress(). The overall output stream would still be valid. But if
callers were expecting input data to exactly map to output data (say the
producer had used flush(COMPRESSOBJ_FLUSH_BLOCK) and was attempting to
map input chunks to output chunks), then the previous behavior would be
wrong. The new behavior is such that output from
flush(COMPRESSOBJ_FLUSH_BLOCK) fed into decompressobj().decompress()
should produce all available compressed input.
* ZstdDecompressor.stream_reader().read() should no longer segfault after
a previous context manager resulted in error.
* ZstdCompressor.compressobj().flush(COMPRESSOBJ_FLUSH_BLOCK) now returns
all data necessary to flush a block. Before, it was possible for the
flush() to not emit all data necessary to fully represent a block. This
would mean decompressors wouldn't be able to decompress all data that had been
fed into the compressor and flush()ed.
New Features
* New module constants BLOCKSIZELOG_MAX, BLOCKSIZE_MAX,
TARGETLENGTH_MAX that expose constants from libzstd.
* New ZstdCompressor.chunker() API for manually feeding data into a
compressor and emitting chunks of a fixed size. Like compressobj(), the
API doesn't impose restrictions on the input or output types for the
data streams. Unlike compressobj(), it ensures output chunks are of a
fixed size. This makes this API useful when the compressed output is being
fed into an I/O layer, where uniform write sizes are useful.
* ZstdCompressor.stream_reader() no longer needs to be used as a context
manager.
* ZstdDecompressor.stream_reader() no longer needs to be used as a context
manager.
* Bundled zstandard library upgraded from 1.3.4 to 1.3.6.
Changes
* Added zstd_cffi.py and NEWS.rst to MANIFEST.in.
* zstandard.__version__ is now defined.
* Upgrade pip, setuptools, wheel, and cibuildwheel packages to latest versions.
* Upgrade various packages used in CI to latest versions. Notably tox (in
order to support Python 3.7).
* Use relative paths in setup.py to appease Python 3.7.
* Added CI for Python 3.7.
Zstandard v1.3.7
perf: slightly better decompression speed on clang (depending on hardware target)
fix: ratio for dictionary compression at levels 9 and 10, reported by @indygreg
build: no longer build backtrace by default in release mode; restrict further automatic mode
build: control backtrace support through build macro BACKTRACE
misc: added man pages for zstdless and zstdgrep, by @samrussell
Zstandard v1.3.6 release is focused on intensive dictionary compression for database scenarios.
This is a new environment we are experimenting. The success of dictionary compression on small data, of which databases tend to store plentiful, led to increased adoption, and we now see scenarios where literally thousands of dictionaries are being used simultaneously, with permanent generation or update of new dictionaries.
== 1.0.0 (2018-05-20)
* *BreakingChange* The XZ module's methods now take any parameters
beyond the IO object as real Ruby keyword arguments rather than
a long argument list.
* *BreakingChange* XZ.decompress_stream now honours Ruby's
external and internal encoding concept instead of just
returning BINARY-tagged strings.
* *BreakingChange* Remove deprecated API on stream reader/writer
class and instead sync the API with Ruby's zlib library
(Ticket #12 by me).
* *BreakingChange* StreamWriter.new and StreamReader.new do not accept
a block anymore. This is part of syncing with Ruby's zlib API.
* *BreakingChange* StreamReader.open and StreamWriter.open always
return the new instance, even if a block is given to the method
(previous behaviour was to return the return value of the block).
This is part of the syncing with Ruby's zlib API.
* *BreakingChange* StreamReader.new and StreamWriter.new as well as
the ::open variants take additional arguments as real Ruby keyword
arguments now instead of a long parameter list plus options hash.
This is different from Ruby's own zlib API as that one takes both
a long parameter list and a hash of additional options. ruby-xz
is meant to follow zlib's semantics mostly, but not as a drop-in
replacement, so this divergence from zlib's API is okay (also
given that it isn't possible to replicate all possible options
1:1 anyway, since liblzma simply accepts different options as
libz). If you've never used these methods' optional arguments,
you should be fine.
* *BreakingChange* Stream#close now returns nil instead of the
number of bytes written. This syncs Stream#close with Ruby's
own IO#close, which also returns nil.
* *BreakingChange* Remove Stream#pos=, Stream#seek, Stream#stat. These
methods irritated the minitar gem, which doesn't expect them to
raise NotImplementedError, but directly to be missing if the object
does not support seeking.
* *BreakingChange* StreamReader and StreamWriter now honour Ruby's
encoding system instead of returning only BINARY-tagged strings.
* *Dependency* Remove dependency on ffi. ruby-xz now uses fiddle from
the stdlib instead.
* *Dependency* Remove dependency on io-like. ruby-xz now implements
all the IO mechanics itself. (Ticket #10 by me)
* *Dependency* Bump required Ruby version to 2.3.0.
* *Fix* libzlma.dylib not being found on OS X (Ticket #15 by
s0nspark).
- perf: minor decompression speed improvement (~+2%) with gcc
- fix : corruption in v1.8.2 at level 9 for files > 64KB under rare
conditions (#560)
- cli : new command --fast, by @jennifermliu
- api : LZ4_decompress_safe_partial() now decodes exactly the nb of
bytes requested (feature request #566)
- build : added Haiku target, by @fbrosson, and MidnightBSD, by @laffer1
- doc : updated documentation regarding dictionary compression
This is based on the decision The NetBSD Foundation made in 2008 to
do so, which was already applied to src.
This change has been applied to code which is likely not in other
repositories.
ok board@, reviewed by riastradh@
1.61 Sat 18 Aug 2018
- File::Find will not untaint [github/ThisUsedToBeAnEmail]
- Prevent from traversing symlinks and parent directories when extracting [github/ppisar]
Changes:
improve q=1 compression on small files
inverse Bazel workspace tree
add rolling-composite-hasher for large-window mode
add tools to download and transform static dictionary data
Changes:
2018-03-15 guidod <guidod@gmx.de>
* fix a number of CVEs reported with special *.zip PoC files
* man-pages are generated with new dbk2man.py - docbook xmlto is optional now
* completing some doc strings while checking the new man-pages to look good
* allow the zziptests.py testsuite to run with an installed /bin path
* try to fix some issues on testing with non-installed binaries on non-linux platfors
* update autotools to allow compiling on some newer Mac / Win machines
* a zip-program is still required for testing, but some errors are gone when not there
* complete the approximation of fnmatch for the test binaries (on platforms without)
* allow windows __mmap.h to be simpler, helping with some problems on MingW
* integrate 'fopen("wb")' from TexLive to be more portable across
* more portability as well for helpers like strnlen being used in the sources
* update doc refs to point to github instead of sf.net
* update the sf.net pages to have a prominent hint on newer github.com location
* release v0.13.69
