Version 7.20.0 (9 February 2010)
Daniel Stenberg (9 Feb 2010)
- When downloading compressed content over HTTP and the app asked libcurl to
automatically uncompress it with the CURLOPT_ENCODING option, libcurl could
wrongly provide the callback with more data than the maximum documented
amount. An application could thus get tricked into badness if the maximum
limit was trusted to be enforced by libcurl itself (as it is documented).
This is further detailed and explained in the libcurl security advisory
20100209 at
http://curl.haxx.se/docs/adv_20100209.html
Daniel Fandrich (3 Feb 2010)
- Changed the Watcom makefiles to make them easier to keep in sync with
Makefile.inc since that can't be included directly.
Yang Tse (2 Feb 2010)
- Symbol CURL_FORMAT_OFF_T now obsoleted, will be removed in a future release,
symbol will not be available when building with CURL_NO_OLDIES defined. Use
of CURL_FORMAT_CURL_OFF_T is preferred since 7.19.0
Daniel Stenberg (1 Feb 2010)
- Using the multi_socket API, it turns out at times it seemed to "forget"
connections (which caused a hang). It turned out to be an existing (7.19.7)
bug in libcurl (that's been around for a long time) and it happened like
this:
The app calls curl_multi_add_handle() to add a new easy handle, libcurl will
then set it to timeout in 1 millisecond so libcurl will tell the app about
it.
The app's timeout fires off that there's a timeout, the app calls libcurl as
we so often document it:
do {
res = curl_multi_socket_action(... TIMEOUT ...);
} while(CURLM_CALL_MULTI_PERFORM == res);
And this is the problem number one:
When curl_multi_socket_action() is called with no specific handle, but only
a timeout-action, it will *only* perform actions within libcurl that are
marked to run at this time. In this case, the request would go from INIT to
CONNECT and return CURLM_CALL_MULTI_PERFORM. When the app then calls libcurl
again, there's no timer set for this handle so it remains in the CONNECT
state. The CONNECT state is a transitional state in libcurl so it reports no
sockets there, and thus libcurl never tells the app anything more about that
easy handle/connection.
libcurl _does_ set a 1ms timeout for the handle at the end of
multi_runsingle() if it returns CURLM_CALL_MULTI_PERFORM, but since the loop
is instant the new job is not ready to run at that point (and there's no
code that makes libcurl call the app to update the timout for this new
timeout). It will simply rely on that some other timeout will trigger later
on or that something else will update the timeout callback. This makes the
bug fairly hard to repeat.
The fix made to adress this issue:
We introduce a loop in lib/multi.c around all calls to multi_runsingle() and
simply check for CURLM_CALL_MULTI_PERFORM internally. This has the added
benefit that this goes in line with my long-term wishes to get rid of the
CURLM_CALL_MULTI_PERFORM all together from the public API.
The downside of this fix, is that the counter we return in 'running_handles'
in several of our public functions then gets a slightly new and possibly
confusing behavior during times:
If an app adds a handle that fails to connect (very quickly) it may just
as well never appear as a 'running_handle' with this fix. Previously it
would first bump the counter only to get it decreased again at next call.
Even I have used that change in handle counter to signal "end of a
transfer". The only *good* way to find the end of a individual transfer
is calling curl_multi_info_read() to see if it returns one.
Of course, if the app previously did the looping before it checked the
counter, it really shouldn't be any new effect.
Yang Tse (26 Jan 2010)
- Constantine Sapuntzakis' and Joshua Kwan's work done in the last four months
relative to the asynchronous DNS lookups, along with with some integration
adjustments I have done are finally committed to CVS.
Currently these enhancements will benefit builds done using c-ares on any
platform as well as Windows builds using the default threaded resolver.
This release does not make generally available POSIX threaded DNS lookups
yet. There is no configure option to enable this feature yet. It is possible
to experimantally try this feature running configure with compiler flags that
make simultaneous definition of preprocessor symbols USE_THREADS_POSIX and
HAVE_PTHREAD_H, as well as whatever reentrancy compiler flags and linker ones
are required to link and properly use pthread_* functions on each platform.
Daniel Stenberg (26 Jan 2010)
- Mike Crowe made libcurl return CURLE_COULDNT_RESOLVE_PROXY when it is the
proxy that cannot be resolved when using c-ares. This matches the behaviour
when not using c-ares.
Bj
- Added a new flag: -J/--remote-header-name. This option tells the
-O/--remote-name option to use the server-specified Content-Disposition
filename instead of extracting a filename from the URL.
Daniel Stenberg (21 Jan 2010)
- Chris Conroy brought support for RTSP transfers, and with it comes 8(!) new
libcurl options for controlling what to get and how to receive posssibly
interleaved RTP data.
Daniel Stenberg (20 Jan 2010)
- As was pointed out on the http-state mailing list, the order of cookies in a
HTTP Cookie: header _needs_ to be sorted on the path length in the cases
where two cookies using the same name are set more than once using
(overlapping) paths. Realizing this, identically named cookies must be
sorted correctly. But detecting only identically named cookies and take care
of them individually is harder than just to blindly and unconditionally sort
all cookies based on their path lengths. All major browsers also already do
this, so this makes our behavior one step closer to them in the cookie area.
Test case 8 was the only one that broke due to this change and I updated it
accordingly.
Daniel Stenberg (19 Jan 2010)
- David McCreedy brought a fix and a new test case (129) to make libcurl work
again when downloading files over FTP using ASCII and it turns out that the
final size of the file is not the same as the initial size the server
reported. This is very common since servers don't take the newline
conversions into account.
Kamil Dudka (14 Jan 2010)
- Suppressed side effect of OpenSSL configure checks, which prevented NSS from
being properly detected under certain circumstances. It had been caused by
strange behavior of pkg-config when handling PKG_CONFIG_LIBDIR. pkg-config
distinguishes among empty and non-existent environment variable in that case.
Daniel Stenberg (12 Jan 2010)
- Gil Weber reported a peculiar flaw with the multi interface when doing SFTP
transfers: curl_multi_fdset() would return -1 and not set and file
descriptors several times during a transfer of a single file. It turned out
to be due to two different flaws now fixed. Gil's excellent recipe helped me
nail this.
Daniel Stenberg (11 Jan 2010)
- Made sure that the progress callback is repeatedly called at a regular
interval even during very slow connects.
- The tests/runtests.pl script now checks to see if the test case that runs is
present in the tests/data/Makefile.am and outputs a notice message on the
screen if not. Each test file has to be included in that Makefile.am to get
included in release archives and forgetting to add files there is a common
mistake. This is an attempt to make it harder to forget.
Daniel Stenberg (9 Jan 2010)
- Johan van Selst found and fixed a OpenSSL session ref count leak:
ossl_connect_step3() increments an SSL session handle reference counter on
each call. When sessions are re-used this reference counter may be
incremented many times, but it will be decremented only once when done (by
Curl_ossl_session_free()); and the internal OpenSSL data will not be freed
if this reference count remains positive. When a session is re-used the
reference counter should be corrected by explicitly calling
SSL_SESSION_free() after each consecutive SSL_get1_session() to avoid
introducing a memory leak.
(http://curl.haxx.se/bug/view.cgi?id=2926284)
Daniel Stenberg (7 Jan 2010)
- Make sure the progress callback is called repeatedly even during very slow
name resolves when c-ares is used for resolving.
Claes Jakobsson (6 Jan 2010)
- Julien Chaffraix fixed so that the fragment part in an URL is not sent
to the server anymore.
Kamil Dudka (3 Jan 2010)
- Julien Chaffraix eliminated a duplicated initialization in singlesocket().
Daniel Stenberg (2 Jan 2010)
- Make curl support --ssl and --ssl-reqd instead of the previous FTP-specific
versions --ftp-ssl and --ftp-ssl-reqd as these options are now used to
control SSL/TLS for IMAP, POP3 and SMTP as well in addition to FTP. The old
option names are still working but the new ones are the ones listed and
documented.
Daniel Stenberg (1 Jan 2010)
- Ingmar Runge enhanced libcurl's FTP engine to support the PRET command. This
command is a special "hack" used by the drftpd server, but even though it is
a custom extension I've deemed it fine to add to libcurl since this server
seems to survive and people keep using it and want libcurl to support
it. The new libcurl option is named CURLOPT_FTP_USE_PRET, and it is also
usable from the curl tool with --ftp-pret. Using this option on a server
that doesn't support this command will make libcurl fail.
I added test cases 1107 and 1108 to verify the functionality.
The PRET command is documented at
http://www.drftpd.org/index.php/Distributed_PASV
Yang Tse (30 Dec 2009)
- Steven M. Schweda improved VMS build system, and Craig A. Berry helped
with the patch and testing.
Daniel Stenberg (26 Dec 2009)
- Renato Botelho and Peter Pentchev brought a patch that makes the libcurl
headers work correctly even on FreeBSD systems before v8.
(http://curl.haxx.se/bug/view.cgi?id=2916915)
Daniel Stenberg (17 Dec 2009)
- David Byron fixed Curl_ossl_cleanup to actually call ENGINE_cleanup when
available.
- Follow-up fix for the proxy fix I did for Jon Nelson's bug. It turned out I
was a bit too quick and broke test case 1101 with that change. The order of
some of the setups is sensitive. I now changed it slightly again to make
sure we do them in this order:
1 - parse URL and figure out what protocol is used in the URL
2 - prepend protocol:// to URL if missing
3 - parse name+password off URL, which needs to know what protocol is used
(since only some allows for name+password in the URL)
4 - figure out if a proxy should be used set by an option
5 - if no proxy option, check proxy environment variables
6 - run the protocol-specific setup function, which needs to have the proxy
already set
Daniel Stenberg (15 Dec 2009)
- Jon Nelson found a regression that turned out to be a flaw in how libcurl
detects and uses proxies based on the environment variables. If the proxy
was given as an explicit option it worked, but due to the setup order
mistake proxies would not be used fine for a few protocols when picked up
from '[protocol]_proxy'. Obviously this broke after 7.19.4. I now also added
test case 1106 that verifies this functionality.
(http://curl.haxx.se/bug/view.cgi?id=2913886)
Daniel Stenberg (12 Dec 2009)
- IMAP, POP3 and SMTP support and their TLS versions (including IMAPS, POP3S
and SMTPS) are now supported. The current state may not yet be solid, but
the foundation is in place and the test suite has some initial support for
these protocols. Work will now persue to make them nice libcurl citizens
until release.
The work with supporting these new protocols was sponsored by
networking4all.com - thanks!
Daniel Stenberg (10 Dec 2009)
- Siegfried Gyuricsko found out that the curl manual said --retry would retry
on FTP errors in the transient 5xx range. Transient FTP errors are in the
4xx range. The code itself only tried on 5xx errors that occured _at login_.
Now the retry code retries on all FTP transfer failures that ended with a
4xx response.
(http://curl.haxx.se/bug/view.cgi?id=2911279)
- Constantine Sapuntzakis figured out a case which would lead to libcurl
accessing alredy freed memory and thus crash when using HTTPS (with
OpenSSL), multi interface and the CURLOPT_DEBUGFUNCTION and a certain order
of cleaning things up. I fixed it.
(http://curl.haxx.se/bug/view.cgi?id=2905220)
Daniel Stenberg (7 Dec 2009)
- Martin Storsjo made libcurl use the Expect: 100-continue header for posts
with unknown size. Previously it was only used for posts with a known size
larger than 1024 bytes.
Daniel Stenberg (1 Dec 2009)
- If the Expect: 100-continue header has been set by the application through
curl_easy_setopt with CURLOPT_HTTPHEADER, the library should set
data->state.expect100header accordingly - the current code (in 7.19.7 at
least) doesn't handle this properly. Martin Storsjo provided the fix!
Yang Tse (28 Nov 2009)
- Added Diffie-Hellman parameters to several test harness certificate files in
PEM format. Required by several stunnel versions used by our test harness.
Daniel Stenberg (28 Nov 2009)
- Markus Koetter provided a polished and updated version of Chad Monroe's TFTP
rework patch that now integrates TFTP properly into libcurl so that it can
be used non-blocking with the multi interface and more. BLKSIZE also works.
The --tftp-blksize option was added to allow setting the TFTP BLKSIZE from
the command line.
Daniel Stenberg (26 Nov 2009)
- Extended and fixed the change I did on Dec 11 for the the progress
meter/callback during FTP command/response sequences. It turned out it was
really lame before and now the progress meter SHOULD get called at least
once per second.
Daniel Stenberg (23 Nov 2009)
- Bjorn Augustsson reported a bug which made curl not report any problems even
though it failed to write a very small download to disk (done in a single
fwrite call). It turned out to be because fwrite() returned success, but
there was insufficient error-checking for the fclose() call which tricked
curl to believe things were fine.
Yang Tse (23 Nov 2009)
- David Byron modified Makefile.dist vc8 and vc9 targets in order to allow
finer granularity control when generating src and lib makefiles.
Yang Tse (22 Nov 2009)
- I modified configure to force removal of the curlbuild.h file included in
distribution tarballs for use by non-configure systems. As intended, this
would get overwriten when doing in-tree builds. But VPATH builds would end
having two curlbuild.h files, one in the source tree and another in the
build tree. With the modification I introduced 5 Nov 2009 this could become
an issue when running libcurl's test suite.
Daniel Stenberg (20 Nov 2009)
- Constantine Sapuntzakis identified a write after close, as the sockets were
closed by libcurl before the SSL lib were shutdown and they may write to its
socket. Detected to at least happen with OpenSSL builds.
- Jad Chamcham pointed out a bug with connection re-use. If a connection had
CURLOPT_HTTPPROXYTUNNEL enabled over a proxy, a subsequent request using the
same proxy with the tunnel option disabled would still wrongly re-use that
previous connection and the outcome would only be badness.
Yang Tse (18 Nov 2009)
- I modified the memory tracking system to make it intolerant with zero sized
malloc(), calloc() and realloc() function calls.
Daniel Stenberg (17 Nov 2009)
- Constantine Sapuntzakis provided another fix for the DNS cache that could
end up with entries that wouldn't time-out:
1. Set up a first web server that redirects (307) to a http://server:port
that's down
2. Have curl connect to the first web server using curl multi
After the curl_easy_cleanup call, there will be curl dns entries hanging
around with in_use != 0.
(http://curl.haxx.se/bug/view.cgi?id=2891591)
- Marc Kleine-Budde fixed: curl saved the LDFLAGS set during configure into
its pkg-config file. So -Wl stuff ended up in the .pc file, which is really
bad, and breaks if there are multiple -Wl in our LDFLAGS (which are in
PTXdist). bug #2893592 (http://curl.haxx.se/bug/view.cgi?id=2893592)
Kamil Dudka (15 Nov 2009)
- David Byron improved the configure script to use pkg-config to find OpenSSL
(and in particular the list of required libraries) even if a path is given
as argument to --with-ssl
Yang Tse (15 Nov 2009)
- I removed enable-thread / disable-thread configure option. These were only
placebo options. The library is always built as thread safe as possible on
every system.
Claes Jakobsson (14 Nov 2009)
- curl-config now accepts '--configure' to see what arguments was
passed to the configure script when building curl.
Daniel Stenberg (14 Nov 2009)
- Claes Jakobsson restored the configure functionality to detect NSS when
--with-nss is set but not "yes".
I think we can still improve that to check for pkg-config in that path etc,
but at least this patch brings back the same functionality we had before.
- Camille Moncelier added support for the file type SSL_FILETYPE_ENGINE for
the client certificate. It also disable the key name test as some engines
can select a private key/cert automatically (When there is only one key
and/or certificate on the hardware device used by the engine)
Yang Tse (14 Nov 2009)
- Constantine Sapuntzakis provided the fix that ensures that an SSL connection
won't be reused unless protection level for peer and host verification match.
I refactored how preprocessor symbol _THREAD_SAFE definition is done.
Kamil Dudka (12 Nov 2009)
- Kevin Baughman provided a fix preventing libcurl-NSS from crash on doubly
closed NSPR descriptor. The issue was hard to find, reported several times
before and always closed unresolved. More info at the RH bug:
https://bugzilla.redhat.com/534176
- libcurl-NSS now tries to reconnect with TLS disabled in case it detects
a broken TLS server. However it does not happen if SSL version is selected
manually. The approach was originally taken from PSM. Kaspar Brand helped me
to complete the patch. Original bug reports:
https://bugzilla.redhat.com/525496https://bugzilla.redhat.com/527771
Yang Tse (12 Nov 2009)
- I modified configure script to make the getaddrinfo function check also
verify if the function is thread safe.
Yang Tse (11 Nov 2009)
- Marco Maggi reported that compilation failed when configured --with-gssapi
and GNU GSS installed due to a missing mutual exclusion of header files in
the Kerberos 5 code path. He also verified that my patch worked for him.
Daniel Stenberg (11 Nov 2009)
- Constantine Sapuntzakis posted bug #2891595
(http://curl.haxx.se/bug/view.cgi?id=2891595) which identified how an entry
in the DNS cache would linger too long if the request that added it was in
use that long. He also provided the patch that now makes libcurl capable of
still doing a request while the DNS hash entry may get timed out.
- Christian Schmitz noticed that the progress meter/callback was not properly
used during the FTP connection phase (after the actual TCP connect), while
it of course should be. I also made the speed check get called correctly so
that really slow servers will trigger that properly too.
Kamil Dudka (5 Nov 2009)
- Dropped misleading timeouts in libcurl-NSS and made sure the SSL socket works
in non-blocking mode.
Yang Tse (5 Nov 2009)
- I removed leading 'curl' path on the 'curlbuild.h' include statement in
curl.h, adjusting auto-makefiles include path, to enhance portability to
OS's without an orthogonal directory tree structure such as OS/400.
Daniel Stenberg (4 Nov 2009)
- I fixed several problems with the transfer progress meter. It showed the
wrong percentage for small files, most notable for <1000 bytes and could
easily end up showing more than 100% at the end. It also didn't show any
percentage, transfer size or estimated transfer times when transferring
less than 100 bytes.
b7789ee748