Bugfix maintenance release. Fixes to hash table expansion now
completely hang all threads very briefly while the hash table
pointers are swapped. Once swapped, it unlocks and uses locks as
normal.
In previous versions, the hash table was switched to a global lock
instead of a map of bucket-locks during expansion. This should be
faster overall with a small latency penalty. It's possible to
presize the hash table with -o hashpower
Fixes
makefile cleanups
Avoid OOM errors when locked items stuck in tail
If clients occasionally fetch many items, more than can fit the
TCP buffers, then hang for a very long period of time, that slab
class could OOM. In older versions this could cause a crash. Since
1.4.20 this will cause OOM errors.
Now, if a locked item lands in the LRU tail, it will be bumped back
to the head and an lrutail_reflocked counter incremented. If you're
concerned about having stuck clients, watch that counter.
Big thanks to Jay Grizzard et all at Box for helping track this
down!
Do it for all packages that
* mention perl, or
* have a directory name starting with p5-*, or
* depend on a package starting with p5-
like last time, for 5.18, where this didn't lead to complaints.
Let me know if you have any this time.
Overview
Just one tiny change to fix a regression causing threads to lock
up and spin max CPU.
1.4.18 and 1.4.19 were affected. 1.4.17 and earlier were not
affected. If you are on .18 or .19 an upgrade to 1.4.20 is strongly
advised.
Thanks to commando.io for reporting the bug initially and putting
up with me being blind for a few weeks.
Fixes
Fix a race condition causing new connections to appear closed,
causing an inifinte loop.
Fixes
Fix endianness detection during configure.
Fixes a performance regression with binary protocol (up to 20%)
Fix rare segfault in incr/decr.
disable tail_repair_time by default.
Likely not needed anymore, and can rarely cause bugs.
use the right hashpower for the item_locks table. Small perf improvement.
Fix crash for LRU crawler while using lock elision (haswell+ processors)
execution of multiple memcached instances. Create home directory in
VARBASE/chroot/memcached for these PID files. Add MESSAGE to explain usage
to user. Usage is simple - set the following in rc.conf:
memcached=YES
memcached_jobs="job0 ... jobN"
memcached_job0_args="-p 11211 -m 64 -c 1024"
...
memcached_jobN_args="-p 1121N -m 64 -c 1024"
Bump PKGREVISION.
Stats conns
New "stats conns" command, which will show you what currently open
connections are up to, how idle they've been, etc. Starttime Hash
Algorithm Selection
The jenkins hash was getting a little long in the tooth, and we
might want to add specific hash algorithms for different platforms
in the future. This makes it selectable in some sense. We've
initially added murmur3 hash to the lineup and that seems to run
a tiny bit faster in some tests.
-o hash_algorithm=murmur3 LRU Crawler
A new background thread emerges! Currently experimental, so the
syntax might change. If you run into bugs please let us know (though
it's been testing fine in torture tests so far).
If you wish to clean your slab classes of items which have been
expired, either one-time or periodically, this will do it with low
impact as a background operation.
Currently it requires kicking off a crawl via manual command:
First, enable the thread: lru_crawler enable or use -o lru_crawler
as a starttime option.
lru_crawler crawl 1,3,5
... would crawl slab classes 1,3,5 looking for expired items to add to the freelist.
This is generally not useful or required, unless you have memory
with very mixed TTLs, you do not fetch items frequently enough or
otherwise cause them to expire, and you don't want items with longer
TTLs block reclaiming expired items, or to be evicted early.
Future uses of the thread should allow examining and purging items
via a plugin interface: IE crawl all items matching some string
and remove them, or count them. It is simple to modify to experiment
with as of now.
See doc/protocol.txt for full explanation of related commands and
counters.
1.4.17:
Another bugfix release along with some minor new features. Most
notable is a potential fix for a crash bug that has plagued the
last few versions. If you see crashes with memcached, please try
this version and let us know if you still see crashes.
The other notable bug is a SASL authentication bypass glitch. If
a client makes an invalid request with SASL credentials, it will
initially fail. However if you issue a second request with bad SASL
credentials, it will authenticate. This has now been fixed.
1.4.16:
A quick bugfix release to get the tree moving again after a long
absence. I don't want to make too many changes at once, so here
are a number of platform and crash fixes, as well as some introspection.
a) refer 'perl' in their Makefile, or
b) have a directory name of p5-*, or
c) have any dependency on any p5-* package
Like last time, where this caused no complaints.
Changes in memcached 1.4.15
Overview
This is a somewhat experimental release which pushes thread
performance even more than before. Since this is a more
experimental release than usual, and contains no other major
fixes or features, we urge some caution for important
deployments. We feel as though it is high quality software, but
please take caution and do slow rollouts or testing. Thanks!
Fixes
* Add some mild thread documentation
* README.md was missing from dist tarball
* Issue 286 : --disable-coverage drops "-pthread" option
* Reduce odds of getting OOM errors in some odd cases
New Features
Thread scalability is much improved for reads, and somewhat
improved for writes. In a pure read-only situation on a dual
socket six core NUMA machine I've tested key fetch rates around
13.6 million keys per second.
More tuning is necessary and you'd get significant lag at that
rate, but that shows the theoretical limit of the locks.
Changes in memcached 1.4.14
Overview
Fixes
* fix compile issue with new GCC's
* Added support for automake-1.12 in autogen.sh
* Use Markdown for README.
* Fixed issue with invalid binary protocol touch command
expiration time (http://code.google.com/p/memcached/issues/detail?id=275)
* Define touch command probe for DTrace support
* Error and exit if we don't have hugetlb support (changes -L behavior)
* update reassign/automove documentation
* Remove USE_SYSTEM_MALLOC define
* slab rebalancing from random class
* split slab rebalance and automove threads
* pre-split slab pages into slab freelists
* Avoid race condition in test during pid creation by blind retrying
New Features
This release mainly features a number of small bugfixes, but
also a change to slab rebalance behavior.
Previously, if you moved a slab page from one slab to another,
you had to wait until that new page was fully used before moving
another one. That wait has been removed, and you can move pages
as fast as the system can ... move them.
A few new features as well:
slabs reassign
slabs reassign -1 15 will pick a page from any slab class and move it to class 15.
slabs automove
slabs automove 2 now enables an ultra aggressive page
reassignment algorithm. On every eviction, it will try to move a
slab page into that class. You should never run this in
production unless you have a very, very good idea of what's
going to happen. For most people who have spurious evictions
everywhere, you'll end up mass evicting random data and hurting
your hit rate. It can be useful to momentarily enable for
emergency situations, or if you have a data access pattern where
evictions should never happen.
This was work we were planning on doing already, but twitter's
rewrite has people presently interested in trying it out. You've
been warned.
Feature and bugfix release. New Touch commands, counters, and a change to connection limit functionality.
*Touch Commands
*Binary Touch/GAT commands were backported from 1.6.
*New GATK/GATKQ commands were added for completeness.
*Finally, an Ascii protocol touch command was also added.
*Fast Connection Limit Handling
*Internal Hash Table
*New Stats: expired_unfetched, evicted_unfetched
1.4.9:
Small bugfix release. Mainly fixing a critical issue where using -c to increase the connection limit was broken in 1.4.8. If you are on 1.4.8, an upgrade is highly recommended.
1.4.10:
This release is focused on thread scalability and performance improvements. This release should be able to feed data back faster than any network card can support as of this writing.
1.4.11:
Memcached 1.4.11. Fixes race conditions and crashes introduced in 1.4.10. Adds the ability to rebalance and reassign slab memory.
*Slab Reassign
*Slab Automove
*New Stats: slab_reassign_running, slabs_moved
1.4.12:
Fix a small number of bugs, mostly in building on different platforms.
1.4.13:
Really tiny release with some important build fixes which were accidentally omitted from 1.4.12.
* Use a monotonically increasing timer
* Immediately expire items when given a negative expiration time
* Fix memcached-tool to print about all slabs
* Properly daemonize memcached for debian
* Don't permanently close UDP listeners on error
* Allow memcached-init to start multiple instances (not recommended)
* Issue 214: Search for network libraries before searching for libevent
* Issue 213: Search for clock_gettime in librt
* Issue 115: accont for CAS in item_size_ok
* Fix incredibly slim race for maxconns handler. Should no longer hang ever
* Issue 183: Reclaim items dead by flush_all
* Issue 200: Don't fire dtrace probe as the last thing in a function
Fixes:
* Properly detect CPU alignment on ARM.
* Remove 1MB assertion.
* More automake versions supported.
* Compiler warning fixes for OpenBSD.
* Potential buffer overflow in vperror.
* Report errors opening pidfiles using vperror.
New Features:
* reclaimed: This stat reports the number of times an entry was stored
using memory from an expired entry.
* sasl_pwdb: Allows memcached to use it's own plantext password file.
(Not configured by pkgsrc yet.)
Critical bug affecting multithreaded builds
Many minor bugfixes, new statistics, minor new features.
Major crash fixes, DTrace support, minor updates.
Minor bugfixes, build support for opensolaris, ipv6 support,
"noreply" mode for many commands, largepage support for solaris.
Made out of memory errors more clear and added eviction/OOM tracking
per slab class.
Many bug and platform fixes since 1.2.2. New threading support for
stat queries. New commands 'append', 'prepend', 'gets', and 'cas'.
2007-04-16 Steven Grimm <sgrimm@facebook.com>
* Command tokenizer performance and cleanliness improvement.
Patch contributed by Paolo Borelli <paolo.borelli@gmail.com>.
2007-04-16 Paul Lindner <lindner@inuus.com>
* Add notes to README about MacOS, libevent and kqueue.
* Windows Patch integration -- part 1, warnings elimination.
2007-04-12 Paul Lindner <lindner@mirth.inuus.com>
* Allow changes to the verbosity level of the server with a new
"verbosity" command and some compiler cleanups.
Patch contributed by Paolo Borelli <paolo.borelli@gmail.com>.
2007-04-08 Paul Lindner <lindner@inuus.com>
* Add cleanup patch from "Tim Yardley" <liquid@haveheart.com> to
clean up source spacing issues, fix -Wall warnings, add some
null checks, adds asserts at the top of each function for any
use of conn *c without checking to see if c is NULL first.
* Also adjust clean-whitespace.pl to clean *.ac files. Add
script to test-suite to test for tabs.
2007-04-04 Paul Lindner <lindner@inuus.com>
* Add clarification of flush_all in the protocol docs
from Elizabeth Mattijsen <liz@dijkmat.nl>
2007-03-31 Paul Lindner <lindner@inuus.com>
* Add patch from Eli Bingham <eli@pandora.com> to
re-enable the -n switch to memcached.
2007-03-20 Paul Lindner <lindner@inuus.com>
* Add patch to collect eviction statistics from
Jean-Francois BUSTARRET <jfbustarret@wat.tv>.
* Updated docs, added new test cases for t/stats.t
2007-03-18 Paul Lindner <lindner@inuus.com>
* Add more test cases using larger buffer sizes up to and greater
than 1MB.
* Remove unused parameter to item_size_ok()
* Use a single printf() in usage()
* Add a failing test for conforming with maximum connections.
2007-03-17
* crash fix from Thomas van Gulick <thomas@partyflock.nl> in
conn_shrink(), passing &ptr, instead of ptr to realloc().
2007-03-05 Paul Lindner <lindner@inuus.com>
* Fix a number of places where (s)printf calls were using unsigned
or signed formats that did not match their arguments.
* Add support for stdbool.h and stdint.h to use the bool and
uint8_t types.
* Major refactoring - move API calls for assoc/items/slabs to
their own individual header files. Add apropriate const and
static declarations as appropriate.
* Avoid type-punning. Do a more efficient realloc inside the
conn_shrink routine.
* Fix overflow bug where uninitialized access to slabclass caused
size-0 mallocs during slab preallocation.
* Use EXIT_SUCCESS/EXIT_FAILURE constants.
* Convert some sprintf calls to snprintf to protect against
buffer overflows.
* Explicitly compare against NULL or zero in many places.
2007-03-05
* Steven Grimm <sgrimm@facebook.com>: Per-object-type stats collection
support. Specify the object type delimiter with the -D command line
option. Turn stats gathering on and off with "stats detail on" and
"stats detail off". Dump the per-object-type details with
"stats detail dump".
2007-03-01
* Steven Grimm <sgrimm@facebook.com>: Fix an off-by-one error in the
multithreaded version's message passing code.
2006-12-23
* fix expirations of items set with absolute expiration times in
the past, before the server's start time. bug was introduced in
1.2.0 with rel_time_t. Thanks to Adam Dixon
<adamtdixon@gmail.com> for the bug report and test case!
changes since 1.1.12:
2006-11-26
* Steven Grimm <sgrimm@facebook.com>: Performance improvements:
Dynamic sizing of hashtable to reduce collisions on very large
caches and conserve memory on small caches.
Only reposition items in the LRU queue once a minute, to reduce
overhead of accessing extremely frequently-used items.
Stop listening for new connections until an existing one closes
if we run out of available file descriptors.
Command parser refactoring: Add a single-pass tokenizer to cut
down on string scanning. Split the command processing into
separate functions for easier profiling and better readability.
Pass key lengths along with the keys in all API functions that
need keys, to avoid needing to call strlen() repeatedly.
2006-11-13
* Iain Wade <iwade@optusnet.com.au>: Fix for UDP responses on
non-"get" commands.
2006-10-13
* Steven Grimm <sgrimm@facebook.com>: New faster hash function.
2006-09-20
* don't listen on UDP by default; more clear message when UDP port
in use
2006-09-09
* release 1.2.0 (along with 1.1.13, which is the more tested branch)
2006-09-02
* Nathan Neulinger <nneul@umr.edu>: fix breakage in expiration code
causing expiration times to not be processed correctly.
2006-08-21
* Nathan Neulinger <nneul@umr.edu>: fix incompatabilities with
unix domain socket support and the UDP code and clean up stale
sockets
2006-08-20
* Nathan Neulinger <nneul@umr.edu>: unix domain socket support
2006-05-03
* Steven Grimm <sgrimm@facebook.com>: big bunch of changes:
big CPU reduction work, UDP-based interface, increased memory
efficiency. (intertwined patch, committed all together)
<http://lists.danga.com/pipermail/memcached/2006-May/002164.html>
or see svn commit logs
2005-08-09
* Elizabeth Mattijsen <liz@dijkmat.nl>: needed a way to flush all
memcached backend servers, but not at exactly the same time (to
reduce load peaks), I've added some simple functionality to the
memcached protocol in the "flush_all" command that allows you to
specify a time at which the flush will actually occur (instead of
always at the moment the "flush_all" command is received).
2005-05-25
* patch from Peter van Dijk <peter@nextgear.nl> to make
stderr unbuffered, for running under daemontools
INSTALLATION_DIRS, as well as all occurrences of ${PREFIX}/man with
${PREFIX}/${PKGMANDIR}.
Fixes PR 35265, although I did not use the patch provided therein.
developer is officially maintaining the package.
The rationale for changing this from "tech-pkg" to "pkgsrc-users" is
that it implies that any user can try to maintain the package (by
submitting patches to the mailing list). Since the folks most likely
to care about the package are the folks that want to use it or are
already using it, this would leverage the energy of users who aren't
developers.
changes since 1.1.11:
2005-04-04
* patch from Don MacAskill <don@smugmug.com> 'flush_all' doesn't
seem to work properly. Basically, if you try to add a key which
is present, but expired, the store fails but the old key is no
longer expired.
2005-01-14
* Date: Thu, 18 Nov 2004 15:25:59 -0600
From: David Phillips <electrum@gmail.com>
Here is a patch to configure.ac and Makefile.am to put the man page in
the correct location. Trying to install the man page from a
subdirectory results in the subdirectory being used in the install
path (it tries to install to doc/memcached.1). This is the correct
thing to do:
- create a Makefile.am in the doc directory that installs the man page
with man_MANS
- modify Makefile.am in the base directory to reference the doc
directory using SUBDIRS
- modify the AC_CONFIG_FILES macro in configure.ac to output the
Makefile in doc
2005-01-14
* pidfile saving support from Lisa Seelye <lisa@gentoo.org>, sent
Jan 13, 2005
2005-01-14
* don't delete libevent events that haven't been added (the deltimer)
patch from Ted Schundler <tschundler@gmail.com>
2004-12-10
* document -M and -r in manpage (Doug Porter <dsp@dsp.name>)
2004-07-22
* fix buffer overflow in items.c with 250 byte keys along with
other info on the same line going into a 256 byte char[].
thanks to Andrei Nigmatulin <anight@monamour.ru>
2004-06-15
* immediate deletes weren't being unlinked a few seconds,
preventing "add" commands to the same key in that time period.
thanks to Michael Alan Dorman <mdorman@debian.org> for the
bug report and demo script.
no longer correct since update to libevent 1.x; it now uses libtool and
generates a shlib.
Remove the offending bl3 line, and bump all dependents' PKGREVISIONs, since
the binary pkg changes for any OS that doesn't have a sufficient builtin
libevent version (or the package has requested a non-builtin version).
memcached is a high-performance, distributed memory object caching
system, generic in nature, but intended for use in speeding up dynamic
web applications by alleviating database load.
