Memcached 1.5.21
Overview
Bugfixes and smaller changes. Thanks to all the external contributors!
Fixes
Adding missing defaults to the --help output
stats: Fix stats delimiter unit tests
Allow compilation with ASAN
restart: add error handling if mmap fails
For text auth token mode, use alternative bcmp implementation
memcached-tool: Fix up tabular output for the 'stats' command.
linux_priv.c: add termios.h include to fix powerpc(64) builds
Update the build documentation in BUILD file
Update documentation for --max-item-size
Fix build issue due to improper pthread_t comparison
Ensure t/whitespace.t test is skipped when outside a memcached git checkout
Allocating large chunk of slabs for FreeBSD.
New Features
configuration option to disable watch commands (-W)
Memcached 1.5.20
Bugfix release.
Security issue: When enabling SASL authentication for binary protocol, enabling UDP mode would allow bypassing SASL. Now refuses to start with both UDP and SASL enabled. Text mode authentication was not vulnerable.
Includes a performance improvement for page rebalancing. Full discussion here: https://github.com/memcached/memcached/pull/524 - in some circumstances, page will be moved between slab classes much faster than before.
Fixes
Security fix: Don't allow UDP with binprot SASL
Remove multiple double-initializations of condition variables and mutexes
Fix data race in assoc_start_expand
Use a proper data type for settings.sig_hup
restart: add error handling when a tag is not found in a metadata file.
doc: Update rfc2629.dtd, use local copy, fix error, and fix warning
doc: Fix out-of-tree build
slab rebalance performance improvements
fix potential deadlock bug in log watcher
Support running tests in out-of-tree build
configure: Fix cross-compilation errors
DTrace build fix
Memcached 1.5.19
This release contains an experimental new protocol extension for the default Text Protocol. This extension includes a flexible set of new commands encompassing features previously unique to the binary protocol, as well as many new updates allowing a reduction of network roundtrips for advanced features.
Fixes
Keep "last access" time up to date in default segmented LRU mode
slow down t/watcher.t test more
Add include stdio.h to restart.c
Reload CA cert in refresh routine
TLS: fix leak of SSL context on accept failure
drop privileges for FreeBSD.
Make memcached setgroups failure non-fatal
restart: use /tmp for restart tests.
restart: add missing msync at close time
Memcached 1.5.18 and newer can recover its cache between restarts. It can restart after upgrades of the binary, most changes in settings, and so on. It now also supports using persistent memory via DAX filesystem mounts.
1.5.17:
Fixes
fix strncpy call in stats conns to avoid ASAN violation
extstore: fix indentation
add error handling when calling dup function
add unlock when item_cachedump malloc failed
extstore: emulate pread(v) for macOS
fix off-by-one in logger to allow CAS commands to be logged.
use strdup for explicitly configured slab sizes
move mem_requested from slabs.c to items.c (internal cleanup)
New Features
add server address to the "stats conns" output
log client connection id with fetchers and mutations
Add a handler for seccomp crashes
1.5.16:
Fixes critical potential segfault/memory corruption bug in 1.5.15 when storing items with client flags of "0". Bug only exists in 1.5.15, and the fix is the only change in this release.
Fixes
* When nsuffix is 0 space for flags hasn't been allocated so don't memcpy them.
1.5.15:
A couple small bugfixes, and two notable changes. An authentication mode for ASCII protocol, and removal of a minor feature flag.
-o inline_ascii_response was a compatability mode which was disabled since 1.5.0; it uses 10-20 more bytes of memory to speed up ASCII responses. Since we switched from snprintf to a fast itoa implementation years ago it was disabled by default. Now the option is gone.
Binary protocol was also not honoring the idle-timeout feature. This should now be fixed.
Fixes
* Speed up incr/decr by replacing snprintf.
* Use correct buffer size for internal URI encoding.
* change some links from http to https
* Fix small memory leak in testapp.c.
* free window_global in slab_automove_extstore.c
* remove inline_ascii_response option
* -Y [filename] for ascii authentication mode
* fix: idle-timeout wasn't compatible with binprot
New Features
-Y [authfile] enables an authentication mode for ASCII protocol. See doc/protocol.txt for the most up to date information. This feature is experimental; there's a small chance it could change based on community feedback. Unlike the SASL authentication for binary protocol, this is built in by default and has no external dependencies. It is also far simpler to use; we get a constant slow trickle of support requests around SASL authentication.
When enabled, UDP and binary protocols are also disabled.
1.5.14:
Bugfix release. No new features. Fix an extstore test failure introduced in 1.5.13. All other bugs are trivial or older.
Fixes
update -h output for -I (max item size)
fix segfault in "lru" command
fix compile error on centos7
extstore: error adjusting page_size after ext_path
extstore: fix segfault if page_count is too high.
close delete + incr item survival race bug
FreeBSD superpages checking.
FreeBSD capabilities support.
memcached-tool dump fix loss of exp value
Add optional feature support to RPM package building
Fix "qw" in "MemcachedTest.pm" so wait_ext_flush is exported properly
Memcached 1.5.12
Overview
Single bugfix release. Fixes a refcount leak that could happen when attempting to run incr/decr against a:
* 0 byte value
* CHUNKED (> 512k) item
* extstore item that has been flushed to disk
Fixes
fix INCR/DECR refcount leak for invalid items
Memcached 1.5.11
Fixes broken test and small extstore optimization.
Fixes
* extstore: balance IO thread queues
* t/lru-maintainer.t: check for WARM item earlier, fixing race condition on some platforms.
Changes for 1.5.6:
From https://github.com/memcached/memcached/wiki/ReleaseNotes156
Main change is disabling UDP to prevent abuse in reflector attack
Fixes
* disable UDP port by default
* systemd instancing support & rpm build improvements
* fix gcc warnings in beta GCC
* fix build with clang
* fix for dtrace compilation on freebsd
New Features
* The RPM specfiles have been greatly improved for systemd installations
Changes for 1.5.5:
From https://github.com/memcached/memcached/wiki/ReleaseNotes155
Fixes
* remove redundant counter/lock from hash table (2% boost for sets)
* limit crawls for metadumper. avoids dumping too much data.
* extstore: revise automove algorithm
* quick fix for slab mover deadlock
* extstore: fix segfault in 'extstore' admin command
New Features
* None.
1.5.4:
New external storage (flash) shim. Still experimental and a little hard to tune, but generally stable and well tested.
Fixes
* make -I argument less position dependent
* external storage base commit
* lru_crawler metadump output ends with "END\r\n"
* fix: -o no_lru_crawler didn't work
New Features
extstore, a cleverly named external storage shim, is an addon for using flash drives to expand cache storage.
1.5.3
Fixes
- Add warning about time on very low TTL's in doc/protocol.txt
- pledge privdropping support for OpenBSD
- make for loop more clear in logger watcher
- fix theoretical leak in process_bin_stat
- fix use of unitialized array in lru_maintainer
- -o no_hashexpand to disable hash table expansion
- fix chunked items set in binprot, read from ascii
New Features
- adds get and touch command for ascii protocol
1.5.2
Fixes
- fix more binary protocol documentation errors.
- fix segfault during 31b -> 32b hash table expand
- don't create hashtables larger than 32bit
- some non-user-facing code changes for supporting future features.
1.5.1
Fixes
- add max_connections stat to 'stats' output
- Drop sockets from obviously malicious command strings (HTTP/)
- stats cachedump: now more likely to show data
- memcached-tool: fix slab Full? column
- fix null pointer ref in logger for bin update cmd
- default to unix sockets for tests, make them much less flaky
- PARALLEL=9 make test -> runs prove in parallel
- fix flaky stats.t test
New Features
- --enable-seccomp compiles in options for strict privilege reduction
in linux. see output of -h for more information.
1.5.0
Fixes
- fix for musl libc: avoid huge stack allocation
New Features
- LRU crawler to background-reclaim memory. Mixed-TTL's and LRU
reordering leaves many holes, making it difficult to properly size
an instance.
- Segmented LRU. HOT/WARM/COLD and background processing should try
harder to keep semi-active items in memory for longer.
- Automated slab rebalancing. Avoiding slab stagnation as objects
change size over time.
- Faster hash table lookups with murmur3 algorithm (though it's been
so long this is now outdated again;)
- Reduce memory requirements per-item by a few bytes here and there
- Immediately close connections when hitting the connection limit,
instead of hanging until a spot opens up.
- Items larger than 512k (by default) are assembled by stacking
multiple chunks together. Now raising the item size above 1m doesn't
drop memory efficiency by spreading out slab classes.
1.4.39
Fixes
- fix for CVE-2017-9951
- save four bytes per item if client flags are 0
New Features
- If client flags are "0", no extra storage is used.
1.4.38
Fixes
- hot_max_age is now hot_max_factor - HOT is now limited to 20% of
COLD's age or 20% of total space, whichever comes first.
- sleep longer between slab move runs (1ms instead of 50us)
- automove script: improve algo, add basic test
- slab_rebal: delete busy items if stuck
- fix LRU maintainer thread slowdown in edge case
- fix rare long background thread pause in hash expansion
1.4.37
Fixes
- LRU crawler: avoid running infinitely.
- fix very old memory leak in ASCII multigets. (when using multiget
gets and keys after the first one are >255 characters)
- remove old slab mover example script.
- fix crash in page mover while using large items
- automover algo python script
- avoid segfault if idle_timeout value is missing.
- fix rare crash in LRU crawler
- sleep more aggressively in some threads
- don't overflow item refcount on get
- fix solaris compilation error
- usability fix for cache_memlimit command
- fix verbose print for idle-kicker
- disable refhang.t test due to flakiness
- fix ordering issue in conn dispatch (prevents potential hangups)
New Features
- LRU crawler scheduling improvements
1.4.36
- Fix refcount leak in LRU bump buf
1.4.35
Fixes
- init.d script status check routine
- Print with more-restricted format string to fix compiler warning
with gcc 7's -Wformat-truncation.
- Display HOT/WARM tail age in stats items
- Active items in HOT' flow to WARM (algorithm fix)
- Moves to WARM requires two hits overall (algorithm fix)
- LRU maintainer performance: per-class sleep scheduling
- Allow limiting the internal LRU crawler run length
- Stop using atomics for item refcount management (performance)
- Make the conn suffix list the same as item list (performance)
- Do LRU-bumps while already holding item lock (performance)
- Reduce add_iov() work for TCP connections (performance)
New Features
- "lru" command for setting LRU parameters at runtime
- Allow switching LRU algo's at runtime
Add -o modern switches to -h
metadump: Fix preventing dumping of class 63
Fix cache_memlimit bug for > 4G values
metadump: ensure buffer is flushed to client before finishing
Number of small fixes/additions to new logging
add logging endpoint for LRU crawler
evicted_active counter for LRU maintainer
stop pushing NULL byte into watcher stream
Scale item hash locks more with more worker threads (minor performance)
Further increase systemd service hardening
Missing necessary header for atomic_inc_64_nv() used in logger.c (solaris)
Fix print format for idle timeout thread
Improve binary sasl security fixes
Fix clang compile error
Widen systemd caps to allow maxconns to increase
Add -X option to disable cachedump/metadump
Don't double free in lru_crawler on closed clients
Fix segfault if metadump client goes away
Automake improvements
Misc documentation fixes
Misc updates to startup scripts
lru_crawler enable blocks until ready (test failure)
Record and report on time spent in listen_disabled (time_in_listen_disabled_us)
Update manpage for -I command. Make it more clear
Fix display of settings.hot_lru_pct in stats settings
No longer edits the output of ps while processing arguments
No longer crashes when failing to give arguments to some start args
Fix memcached unable to bind to an ipv6 address
No longer attempts bind to same interface more than once
fixed libevent version check: add the missing 1.0.x version check
fix off-by-one in LRU crawler, causing rare segfault
remove another invalid assert(), fixes clang and pedantic compilation
Issues found with existing distfiles:
distfiles/eclipse-sourceBuild-srcIncluded-3.0.1.zip
distfiles/fortran-utils-1.1.tar.gz
distfiles/ivykis-0.39.tar.gz
distfiles/enum-1.11.tar.gz
distfiles/pvs-3.2-libraries.tgz
distfiles/pvs-3.2-linux.tgz
distfiles/pvs-3.2-solaris.tgz
distfiles/pvs-3.2-system.tgz
No changes made to these distinfo files.
Otherwise, existing SHA1 digests verified and found to be the same on
the machine holding the existing distfiles (morden). All existing
SHA1 digests retained for now as an audit trail.
FreeBSD with clang: format specifies type 'int' but the argument has type 'long'
This seems to be fallout from the time_t change patches. Defuzz patches
while we're here.
Bugfix maintenance release. Fixes to hash table expansion now
completely hang all threads very briefly while the hash table
pointers are swapped. Once swapped, it unlocks and uses locks as
normal.
In previous versions, the hash table was switched to a global lock
instead of a map of bucket-locks during expansion. This should be
faster overall with a small latency penalty. It's possible to
presize the hash table with -o hashpower
Fixes
makefile cleanups
Avoid OOM errors when locked items stuck in tail
If clients occasionally fetch many items, more than can fit the
TCP buffers, then hang for a very long period of time, that slab
class could OOM. In older versions this could cause a crash. Since
1.4.20 this will cause OOM errors.
Now, if a locked item lands in the LRU tail, it will be bumped back
to the head and an lrutail_reflocked counter incremented. If you're
concerned about having stuck clients, watch that counter.
Big thanks to Jay Grizzard et all at Box for helping track this
down!
Do it for all packages that
* mention perl, or
* have a directory name starting with p5-*, or
* depend on a package starting with p5-
like last time, for 5.18, where this didn't lead to complaints.
Let me know if you have any this time.
Overview
Just one tiny change to fix a regression causing threads to lock
up and spin max CPU.
1.4.18 and 1.4.19 were affected. 1.4.17 and earlier were not
affected. If you are on .18 or .19 an upgrade to 1.4.20 is strongly
advised.
Thanks to commando.io for reporting the bug initially and putting
up with me being blind for a few weeks.
Fixes
Fix a race condition causing new connections to appear closed,
causing an inifinte loop.
Fixes
Fix endianness detection during configure.
Fixes a performance regression with binary protocol (up to 20%)
Fix rare segfault in incr/decr.
disable tail_repair_time by default.
Likely not needed anymore, and can rarely cause bugs.
use the right hashpower for the item_locks table. Small perf improvement.
Fix crash for LRU crawler while using lock elision (haswell+ processors)
execution of multiple memcached instances. Create home directory in
VARBASE/chroot/memcached for these PID files. Add MESSAGE to explain usage
to user. Usage is simple - set the following in rc.conf:
memcached=YES
memcached_jobs="job0 ... jobN"
memcached_job0_args="-p 11211 -m 64 -c 1024"
...
memcached_jobN_args="-p 1121N -m 64 -c 1024"
Bump PKGREVISION.
Stats conns
New "stats conns" command, which will show you what currently open
connections are up to, how idle they've been, etc. Starttime Hash
Algorithm Selection
The jenkins hash was getting a little long in the tooth, and we
might want to add specific hash algorithms for different platforms
in the future. This makes it selectable in some sense. We've
initially added murmur3 hash to the lineup and that seems to run
a tiny bit faster in some tests.
-o hash_algorithm=murmur3 LRU Crawler
A new background thread emerges! Currently experimental, so the
syntax might change. If you run into bugs please let us know (though
it's been testing fine in torture tests so far).
If you wish to clean your slab classes of items which have been
expired, either one-time or periodically, this will do it with low
impact as a background operation.
Currently it requires kicking off a crawl via manual command:
First, enable the thread: lru_crawler enable or use -o lru_crawler
as a starttime option.
lru_crawler crawl 1,3,5
... would crawl slab classes 1,3,5 looking for expired items to add to the freelist.
This is generally not useful or required, unless you have memory
with very mixed TTLs, you do not fetch items frequently enough or
otherwise cause them to expire, and you don't want items with longer
TTLs block reclaiming expired items, or to be evicted early.
Future uses of the thread should allow examining and purging items
via a plugin interface: IE crawl all items matching some string
and remove them, or count them. It is simple to modify to experiment
with as of now.
See doc/protocol.txt for full explanation of related commands and
counters.
1.4.17:
Another bugfix release along with some minor new features. Most
notable is a potential fix for a crash bug that has plagued the
last few versions. If you see crashes with memcached, please try
this version and let us know if you still see crashes.
The other notable bug is a SASL authentication bypass glitch. If
a client makes an invalid request with SASL credentials, it will
initially fail. However if you issue a second request with bad SASL
credentials, it will authenticate. This has now been fixed.
1.4.16:
A quick bugfix release to get the tree moving again after a long
absence. I don't want to make too many changes at once, so here
are a number of platform and crash fixes, as well as some introspection.
a) refer 'perl' in their Makefile, or
b) have a directory name of p5-*, or
c) have any dependency on any p5-* package
Like last time, where this caused no complaints.
Changes in memcached 1.4.15
Overview
This is a somewhat experimental release which pushes thread
performance even more than before. Since this is a more
experimental release than usual, and contains no other major
fixes or features, we urge some caution for important
deployments. We feel as though it is high quality software, but
please take caution and do slow rollouts or testing. Thanks!
Fixes
* Add some mild thread documentation
* README.md was missing from dist tarball
* Issue 286 : --disable-coverage drops "-pthread" option
* Reduce odds of getting OOM errors in some odd cases
New Features
Thread scalability is much improved for reads, and somewhat
improved for writes. In a pure read-only situation on a dual
socket six core NUMA machine I've tested key fetch rates around
13.6 million keys per second.
More tuning is necessary and you'd get significant lag at that
rate, but that shows the theoretical limit of the locks.
Changes in memcached 1.4.14
Overview
Fixes
* fix compile issue with new GCC's
* Added support for automake-1.12 in autogen.sh
* Use Markdown for README.
* Fixed issue with invalid binary protocol touch command
expiration time (http://code.google.com/p/memcached/issues/detail?id=275)
* Define touch command probe for DTrace support
* Error and exit if we don't have hugetlb support (changes -L behavior)
* update reassign/automove documentation
* Remove USE_SYSTEM_MALLOC define
* slab rebalancing from random class
* split slab rebalance and automove threads
* pre-split slab pages into slab freelists
* Avoid race condition in test during pid creation by blind retrying
New Features
This release mainly features a number of small bugfixes, but
also a change to slab rebalance behavior.
Previously, if you moved a slab page from one slab to another,
you had to wait until that new page was fully used before moving
another one. That wait has been removed, and you can move pages
as fast as the system can ... move them.
A few new features as well:
slabs reassign
slabs reassign -1 15 will pick a page from any slab class and move it to class 15.
slabs automove
slabs automove 2 now enables an ultra aggressive page
reassignment algorithm. On every eviction, it will try to move a
slab page into that class. You should never run this in
production unless you have a very, very good idea of what's
going to happen. For most people who have spurious evictions
everywhere, you'll end up mass evicting random data and hurting
your hit rate. It can be useful to momentarily enable for
emergency situations, or if you have a data access pattern where
evictions should never happen.
This was work we were planning on doing already, but twitter's
rewrite has people presently interested in trying it out. You've
been warned.
