This flag should be set for packages that import pkg_resources
and thus need setuptools after the build step.
Set this flag for packages that need it and bump PKGREVISION.
3.8.0: 2020-10-29
=================
* Better handling of invalid bytecode magic
* Support running from 3.9 and 3.10 although we do not support those bytecodes
* Redo version comparisons using tuples instead of floats. This is needed for Python 3.10
* Split out into 3 branches so that the master branch can assume Python 3.6+ conventions, especially type annotations
* Source Fragment fixes
* Lambda-bug fixes 360
* Bug fixes
3.7.4:
Fragment parsing was borked. This means deparsing in trepan2/trepan3k was broken
3.7+: narrow precedence for call tatement
del_stmt -> delete to better match Python AST
3.8+ Add another forelsestmt (found only in a loop)
3.8+ Add precedence on walrus operator
More files blackened
bump min xdis version
3.7.3:
Mostly small miscellaneous bug fixes
__doc__ = DocDescr() from test_descr.py was getting confused as a docstring.
detect 2.7 exchandler range better
Add for .. else reduction checks on 2.6 and before
Add reduce check for 2.7 augmented assign
Add VERSION in a pydoc-friendly way
3.7.2:
Use newer xdis
Docstrings (again) which were broken again on earlier Python
Fix 2.6 and 2.7 decompilation bug in handling "list if" comprehensions
3.7.1:
Released to pick up new xdis version which has fixes to read bytestings better on 3.x
Handle 3.7+ "else" branch removal adAs seen in _cmp() of python3.8/distutils/version.py with optimization -O2
3.6+ "with" and "with .. as" grammar improvements
ast-check for "for" loop was missing some grammar rules
3.6.6: 2020-4-20 Love in the time of Cholera
The main reason for this release is an incompatablity bump in xdis which handles 3.7 SipHash better.
Go over "yield" as an expression precidence
Some small alignment with code in decompyle3 for "or" and "and" was done
3.6.5:
Back port some of the changes in decompile3 here which mostly helps 3.7 and 3.8 decompilation, although this may also help 3.6ish versions too.
Handle nested async for in for... and better async comprehension detection via xdis. Still more work is needed.
include token number in listings when -g and there is a parser error
remove unneeded Makefiles now that remake 4.3+1.5dbg is a thing that has -c
Bug in finding annotations in functions with docstrings
Fix bug found by 2.4 sre_parse.py testing
Fix transform module's ifelseif bugs
Fix bug in 3.0 name module detection
Fix docstring detection
3.6.4:
plateau
The main focus in this release was fix some of the more glaring problems creapt in from the last release due to that refactor.
uncompyle6 code is at a plateau where what is most needed is a code refactoring. In doing this, until everything refactored and replaced, decomplation may get worse.
Therefore, this release largely serves as a checkpoint before more major upheaval.
The upheaval, in started last release, I believe the pinnicle was around c90ff51 which wasn't a release. I suppose I should tag that.
After c90ff5, I started down the road of redoing control flow in a more comprehensible, debuggable, and scalable way. See The Control Flow Mess.
The bulk of the refactoring going on in the decompyle3 project, but I try to trickle down the changes.
It is tricky because the changes are large and I have to figure decompose things so that little testable pieces can be done. And there is also the problem that what is in decompyle3 is incomplete as well.
Other than control flow, another change that will probably happen in the next release is to redo the grammar for lambda expressions. Right now, we treat them as Python statements, you know, things with compound statements in them. But lambdas aren't that. And so there is hackery to paper over difference making a statement out of an expression the wrong thing to do. For example, a return of an "and" expression can be expressed as nested "if" statements with return inside them, but the "if" variant of the bytecode is not valid in a lambda.
In the decompyle3 code, I've gone down the road making the grammar goal symbol be an expression. This also offers the opportunity to split the grammar making parsing inside lambda not only more reliable because the wrong choices don't exist, but also simpler and faster because all those rules just need don't need to exist in parsing.
I cringe in thinking about how the code has lived for so long without noticing such a simple stupidity, and lapse of sufficient thought.
3.6.3:
Martin and Susanne
Of late, every release fixes major gaps and embarrassments of the last release....
And in some cases, like this one, exposes lacuna and rot.
I now have [control] flow under control, even if it isn't the most optimal way.
I now have greatly expanded automated testing.
On the most recent Python versions I regularly decompile thousands of Python programs that are distributed with Python. when it is possible, I then decompile Python's standard test suite distributed with Python and run the decompiled source code which basically checks itself. This amounts to about 250 test programs per version. This is in addition to the 3 CI testing services which do different things.
Does this mean the decompiler works perfectly? No. There are still a dozen or so failing programs, although the actual number of bugs is probably smaller though.
However, in perparation of a more major refactoring of the parser grammar, this release was born.
In many cases, decompilation is better. But there are some cases where decompilation has gotten worse. For lack of time (and interest) 3.0 bytecode suffered a hit. Possibly some code in the 3.x range did too. In time and with cleaner refactored code, this will come back.
Commit c90ff51 was a local maxiumum before, I started reworking the grammar to separate productions that were specific to loops versus those that are not in loops.
In the middle of that I added another grammar simplication to remove singleton productions of the form sstmts-> stmts. These were always was a bit ugly, and complicated output.
At any rate if decompilation fails, you can try c90ff51. Or another decompiler. unpyc37 is pretty good for 3.7. wibiti uncompyle2 is great for 2.7. pycdc is mediocre for Python before 3.5 or so, and not that good for the most recent Python. Generally these programs will give some sort of answer even if it isn't correct.
decompyle3 isn't that good for 3.7 and worse for 3.8, but right now it does things no other Python decompiler like unpyc37 or pycdc does. For example, decompyle3 handles variable annotations. As always, the issue trackers for the various programs will give you a sense for what needs to be done. For now, I've given up on reporting issues in the other decompilers because there are already enough issues reported, and they are just not getting fixed anyway.
3.6.2:
Yet again the focus has been on just fixing bugs, mostly geared in the
later 3.x range. To get some sense what sill needs fixing, consult
test/stdlib/runtests.sh. And that only has a portion of what's known.
make_function.py has gotten so complex that it was split out into 3 parts
to handle different version ranges: Python <3, Python 3.0..3.6 and Python 3.7+.
An important fix is that we had been dropping docstrings in Python 3 code as a result
of a incomplete merge from the decompile3 base with respect to the transform phase.
Also important (at least to me) is that we can now handle 3.6+
variable type annotations. Some of the decompile3 code uses that in
its source code, and I now use variable annotations in conjunction
with mypy in some of my other Python projects
Code generation for imports, especially where the import is dotted
changed a bit in 3.7; with this release are just now tracking that
change better. For this I've added pseudo instruction
IMPORT_NAME_ATTR, derived from the IMPORT_NAME instruction, to
indicate when an import contains a dotted import. Similarly, code for
3.7 import .. as is basically the same as from .. import, the
only difference is the target of the name changes to an "alias" in the
former. As a result, the disambiguation is now done on the semantic
action side, rathero than in parsing grammar rules.
Some small specific fixes:
3.7+ some chained compare parsing has been fixed. Other remain.
better if/else rule checking in the 3.4 and below range.
3.4+ keyword-only parameter handling was fixed more generally
3.3 .. 3.5 keyword-only parameter args in lambda was fixed
3.6.1:
Overall, as in the past, the focus has been on just fixing bugs, more geared
in the later 3.x range. Handling "async for/with" in 3.8+ works better.
Numerous bugs around handling lambda with keyword-only and * args in the
3.0-3.8 have been fixed. However many still remain.
binary_expr and unary_expr have been renamed to bin_op and
unary_op to better correspond the Python AST names.
Some work was done Python 3.7+ to handle and better; less was done
along the lines of handling or. Much more is needed to improve
parsing stability of 3.7+. More of what was done with and needs to
be done with or and this will happen first in the "decompyle3"
project.
Later this will probably be extended backwards to handle the 3.6-
versions better. This however comes with a big decompilation speed
penalty. When we redo control flow this should go back to normal, but
for now, accuracy is more important than speed.
Another assert transform rule was added. Parser rules to distingish
try/finally in 3.8 were added and we are more stringent about what
can be turned into an assert. There was some grammar cleanup here
too.
A number of small bugs were fixed, and some administrative changes to
make make check-short really be short, but check more throughly what
it checks. minimum xdis version needed was bumped to include in the
newer 3.6-3.9 releases. See the ChangeLog for details.
3.6.0:
The main focus in this release was more accurate decompilation especially for 3.7 and 3.8. However there are some improvments to Python 2.x as well, including one of the long-standing problems of detecting the difference between try ... and try else ....
With this release we now rebase Python 3.7 on off of a 3.7 base; This is also as it is (now) in decompyle3. This facilitates removing some of the cruft in control-flow detection in the 2.7 uncompyle2 base.
Alas, decompilation speed for 3.7 on is greatly increased. Hopefull this is temporary (cough, cough) until we can do a static control flow pass.
Finally, runing in 3.9-dev is tolerated. We can disassemble, but no parse tables yet.
3.5.1:
Pypy 3.3, 3.5, 3.6, and 3.6.9 support
Improve 3.0 decompilation
- no parse errors on stlib bytecode. However accurate translation in
- control-flow and and/or detection needs work
Remove extraneous iter() in "for" of list comprehension
"for" block without a POP_BLOCK and confusing JUMP_BACK for CONTINUE.
Fix unmarshal incompletness detected in Pypy 3.6
Miscellaneous bugs fixed
3.5.0:
Fix fragment bugs
- missing RETURN_LAST introduced when adding transformation layer
- more parent entries on tokens
Preliminary support for decompiling Python 1.0, 1.1. 1.2 and 1.6
- Newer xdis version needed
3.4.0:
The main change is to add a tree-transformation phase. This simplifies the
code a little and allows us to turn if ...: raise AssertionError into
assert, and many if ..: else if ... into if ... elif ..
Use options --show=before and --show=after to see the before the tree transformation phase and after the tree transformation phase.
3.3.5:
Handle annotation args in Python 3.x
Fix vararg and function signatures in 3.x
Some 3.x < 3.6 while (1)/if fixes — others remain
Start reinstating else if -> elif
LOAD_CONST -> LOAD_CODE where appropriate
option --weak-verify is now --syntax-verify
code cleanups, start using black to reformat text
3.3.4:
Major work was done by x0ret to correct function signatures and include annotation types
Handle Python 3.6 STORE_ANNOTATION
Friendlier assembly output
LOAD_CONST replaced by LOAD_STR where appropriate to simplify parsing and improve clarity
remove unneeded parenthesis in a generator expression when it is the single argument to the function
Bug in noting an async function
Handle unicode docstrings and fix docstring bugs
Add uncompyle6 command-line short option -T as an alternate for --tree+
Some grammar cleanup
3.3.3:
As before, decomplation bugs fixed. The focus has primarily been on Python 3.7. But with this release, releases will be put on hold,as a better control-flow detection is worked on . Tis has been needed for a while, and is long overdue. It will probably also take a while to get done as good as what we have now.
However this work will be done in a new project decompyle3. In contrast to uncompyle6 the code wil be written assuming a modern Python 3, e.g. 3.7. It is originally intended to decompile Python version 3.7 and greater.
A number of Python 3.7+ chained comparisons were fixed
Revise Python 3.6ish format string handling
Go over operator precedence, e.g. for AST IfExp
release-python-3.3.2
As before, lots of decomplation bugs fixed. The focus has primarily
been on Python 3.6. We can now parse the entire 3.6.8 Python library
and verify that without an error. The same is true for 3.5.8. A number
of the bugs fixed though are not contained to these versions. In fact
some span back as far as 2.x
But as before, many more remain in the 3.7 and 3.8 range which will
get addressed in future releases
Pypy 3.6 support was started. Pypy 3.x detection fixed (via xdis)
release-3.3.1
Lots of decomplation bugs, especially in the 3.x series fixed. Don't worry though, many more remain.
* Add annotation return values in 3.6+
* Fix 3.6+ lambda parameter handling decompilation
* Fix 3.7+ chained comparision decompilation
* split out semantic-action customization into more separate files
* Add 3.8 try/else
* Fix 2.7 generator decompilation
* Fix some parser failures fixes in 3.4+ using test_pyenvlib
* Add more run tests
release-3.3.0
First cut at Python 3.8 (many bugs remain)
Reinstate -c | --compile (compile before disassembly) option
The usual smattering of bug and doc fixes
3.2.6:
Bug Fixes
* Python 3.x bytecode confusing "try/else" with "try" in a loop,
* Python 3 bug in not detecting end bounds of an "if" ... "elif",
* Comma placement in 3.6 and 3.7 **kwargs,
* Fix "if" return boundary in 3.6+,
* 2.7 can have two JUMP_BACKs at the end of a while loop
Pull Requests
* Better "assert" statement detemination in Python 2.7
* Python 3.7 testing
* Run more f-string tests on Python 3.7
* support utf-8 chars in Python 3 sourcecode
uncompyle6 3.2.5:
- 3.7.2 Remove deprecation warning on regexp string that isn't raw
- main.main() parameter codes is not used - note that
- Improve Python 3.6+ control flow detection
- More complete fragment instruction annotation for imports
A native Python cross-version decompiler and fragment decompiler. The successor
to decompyle, uncompyle, and uncompyle2.
uncompyle6 translates Python bytecode back into equivalent Python source code.
It accepts bytecodes from Python version 1.3 to version 3.7, spanning over 22
years of Python releases. We include Dropbox's Python 2.5 bytecode and some
PyPy bytecode.
