This will be the last release of chardet to support Python 2.7.
chardet 5.0 will only support 3.6+
Major Changes
This release is multiple years in the making, and provides some
quality of life improvements to chardet. The primary user-facing
changes are:
Single-byte charset probers now use nested dictionaries under
the hood, so they are usually a little faster than before. (See
#121 for details)
The CharsetGroupProber class now properly short-circuits when
one of the probers in the group is considered a definite match.
This lead to a substantial speedup.
There is now a chardet.detect_all function that returns a list
of possible encodings for the input with associated confidences.
We have dropped support for Python 2.6, 3.4, and 3.5 as they
are all past end-of-life.
The changes in this release have also laid the groundwork for
retraining the models to make them more accurate, and to support
some more encodings/languages (see #99 for progress). This is our
main focus for chardet 5.0 (beyond dropping Python 2 support).
This minor bugfix release just fixes some packaging and documentation issues:
* Fix issue with setup.py where pytest_runner was always being installed.
* Make sure test.py is included in the manifest
* Fix a bunch of old URLs in the README and other docs.
* Update documentation to no longer imply we test/support Python 3 versions before 3.3
chardet 3.0.2
Fixes an issue where detect would sometimes return None instead of a dict with the keys encoding, language, and confidence (Issue #113, PR #114).
chardet 3.0.1
This bugfix release fixes a crash in the EUC-TW prober when it encountered certain strings (Issue #67).
chardet 3.0.0
This release is long overdue, but still mostly serves as a placeholder
for the impending 4.0.0 release, which will have retrained models
for better accuracy. For now, this release will get the following
improvements up on PyPI:
Added support for Turkish ISO-8859-9 detection (PR #41, thanks @queeup)
Commented out large unused sections of Big5 and EUC-KR tables to save memory (8bc4b89)
Removed Python 3.2 from testing, but add 3.4 - 3.6
Ensure that stdin is open with mode 'rb' for chardetect CLI. (PR #38, thanks @lpsinger)
Fixed chardetect crash with non-ascii file names (PR #39, thanks @nkanaev)
Made naming conventions more Pythonic throughout (no more mTypicalPositiveRatio, and instead typical_positive_ratio)
Modernized test scripts and infrastructure so we've got Travis testing and all that stuff
Rename filter_without_english_words to filter_international_words and make it match current Mozilla implementation (PR #44, thanks @rsnair2)
Updated filter_english_letters to match C implementation (c665459)
Temporarily disabled Hungarian ISO-8859-2 and Windows-1250 detection because it is very inaccurate (da6c0a0)
Allow CLI sub-package to be importable (PR #55)
Add a hypotheis-based test (PR #66, thanks @DRMacIver)
Strip endianness from UTF with BOM predictions so that the encoding can be passed directly to bytes.decode() (PR #73, thanks @snoack)
Fixed broken links in docs (PR #90, thanks @roskakori)
Added early exit to chardetect when encoding is detected instead of looping through entire file (PR #103, thanks @jpz)
Use bytearray objects internally instead of wrap_ord calls, which provides a nice performance boost across the board (PR #106)
Add language property to probers and UniversalDetector results (PR #180)
Mark the 5 known test failures as such so we can have more useful Travis build results in the meantime (d588407)
Problems found with existing distfile:
distfiles/libiconv-1.13-cp932.patch.gz
No changes made to the libiconv distinfo file.
Otherwise, existing SHA1 digests verified and found to be the same on
the machine holding the existing distfiles (morden). All existing
SHA1 digests retained for now as an audit trail.
either because they themselves are not ready or because a
dependency isn't. This is annotated by
PYTHON_VERSIONS_INCOMPATIBLE= 33 # not yet ported as of x.y.z
or
PYTHON_VERSIONS_INCOMPATIBLE= 33 # py-foo, py-bar
respectively, please use the same style for other packages,
and check during updates.
Use versioned_dependencies.mk where applicable.
Use REPLACE_PYTHON instead of handcoded alternatives, where applicable.
Reorder Makefile sections into standard order, where applicable.
Remove PYTHON_VERSIONS_INCLUDE_3X lines since that will be default
with the next commit.
Whitespace cleanups and other nits corrected, where necessary.
2.0.1 (2009-11-10)
* Migrated to http://code.google.com/p/chardet
* Fixed minor typo in documentation
* Synchronized version numbers of Python 2 and Python 3 versions
on some platforms that lacked shared library support in the past. The
list hasn't been maintained at all and the gain is very limited, so just
get rid of it.
