4.3.0:
- Added support for broadcasting using a regular expression pattern
or a glob pattern to multiple Pidboxes.
- Added support for PEP 420 namespace packages.
- Added :setting:acks_on_failure_or_timeout as a setting instead of
a task only option.
- Added the :signal:task_received signal.
- Fixed a crash of our CLI that occurred for everyone using Python < 3.6.
- Fixed a crash that occurred when using the Redis result backend
while the :setting:result_expires is set to None.
- Added support the DNS seedlist connection format for the MongoDB result backend.
- Bump the minimum eventlet version to 0.24.1.
- Replace the msgpack-python package with msgpack.
- Allow scheduling error handlers which are not registered tasks in the current
worker.
- Additional fixes and enhancements to the SSL support of
the Redis broker and result backend.
4.2.1:
- **Result Backend**: Fix deserialization of exceptions that are present in the producer codebase but not in the consumer codebase.
- **Message Protocol Compatibility**: Fix error caused by an invalid (None) timelimit value in the message headers when migrating messages from 3.x to 4.x.
- **Result Backend**: Fix serialization of exception arguments when exception arguments are not JSON serializable by default.
- **Worker**: Fixed multiple issues with rate limited tasks
Maintain scheduling order.
Fix possible scheduling of a :class:celery.worker.request.Request with the wrong :class:kombu.utils.limits.TokenBucket which could cause tasks' rate limit to behave incorrectly.
Fix possible duplicated execution of tasks that were rate limited or if ETA/Countdown was provided for them.
- **Worker**: Defensively handle invalid timelimit header values in requests.
4.2.0:
- **Task**: Add ignore_result as task execution option
- **Redis Result Backend**: Do not create PubSub subscriptions when results are ignored
- **Redis Result Backend**: Result consumer always unsubscribes when task state is ready
- **Development/Testing**: Add docker-compose and base Dockerfile for development
- **Documentation/Sphinx**: Teach autodoc to document tasks if undoc-members is not set
- **Documentation/Sphinx**: Put back undoc-members option in sphinx test
- **Documentation/Sphinx**: Sphinx autodoc picks up tasks automatically only if undoc-members is set
- **Task**: Fix shadow_name issue when using previous version Task class
- **Task**: Add support for bound tasks as link_error parameter
- **Deployment**: Add a command line option for setting the Result Backend URL
- **CI**: Enable pip cache in appveyor build
- **Concurrency/Asynpool**: Fix errno property name shadowing.
- **DynamoDB Backend**: Configurable endpoint URL
- **Timezones**: Correctly detect UTC timezone and timezone from settings
- **Control**: Cleanup the mailbox's producer pool after forking
- **Documentation**: Start Celery and Celery Beat on Azure WebJob
- **Celery Beat**: Schedule due tasks on startup, after Beat restart has occurred
- **Worker**: Use absolute time when task is accepted by worker pool
- **Canvas**: Propagate arguments to chains inside groups
- **Canvas**: Fix Task.replace behavior in nested chords
- **Installation**: Pass python_requires argument to setuptools
- **Message Protocol Compatibility**: Handle "hybrid" messages that have moved between Celery versions
- **Canvas**: request on_timeout now ignores soft time limit exception
- **Redis Result Backend**: Integration test to verify PubSub unsubscriptions
- **Message Protocol Properties**: Allow the shadow keyword argument and the shadow_name method to set shadow properly
- **Canvas**: Run chord_unlock on same queue as chord body
- **Canvas**: Support chords with empty header group
- **Timezones**: make astimezone call in localize more safe
- **Canvas**: Fix length-1 and nested chords
- **CI**: Run Openstack Bandit <https://pypi.org/project/bandit/1.0.1/>_ in Travis CI in order to detect security issues.
- **CI**: Run isort <https://github.com/timothycrosley/isort>_ in Travis CI in order to lint Python **import** statements.
- **Canvas**: Resolve TypeError on .get from nested groups
- **CouchDB Backend**: Correct CouchDB key string type for Python 2/3 compatibility
- **Group Result**: Fix current_app fallback in GroupResult.restore()
- **Consul Backend**: Correct key string type for Python 2/3 compatibility
- **Group Result**: Correctly restore an empty GroupResult
- **Result**: Disable synchronous waiting for sub-tasks on eager mode
- **Celery Beat**: Detect timezone or Daylight Saving Time changes
- **Canvas**: Fix append to an empty chain.
- **Task**: Allow shadow to override task name in trace and logging messages.
- **Documentation/Sphinx**: Fix getfullargspec Python 2.x compatibility in contrib/sphinx.py
- **Documentation**: Updated installation instructions for SQS broker
- **Celery Beat**: Better equality comparison for ScheduleEntry instances
- **Task**: Adding 'shadow' property to as_task_v2
- Try to import directly, do not use deprecated imp method
- **Task**: Enable kwargsrepr and argsrepr override for modifying task argument representation
- **Result Backend**: Add Redis Sentinel backend
- Use unique time values for Collections/LimitedSet
- **CI**: Report coverage for all result backends.
- **Django**: Use Django DB max age connection setting
- **Canvas**: Properly take into account chain tasks link_error
- **Canvas**: Allow to create group with single task
- **Canvas**: Copy dictionary parameter in chord.from_dict before modifying
- **Results Backend**: Add Cassandra options
- **Worker**: Apply rate limiting for tasks with ETA
- **Celery Beat**: support scheduler entries without a schedule
- **SQS Broker**: Updated SQS requirements file with correct boto3 version
- Remove unused code from _create_app contextmanager
- **Group Result**: Modify GroupResult.as_tuple() to include parent
- **Beat**: Set default scheduler class in beat command.
- **Worker**: Retry signal receiver after raised exception
- **Task**: Allow custom Request class for tasks
- **Django**: Django fixup should close all cache backends
- **Deployment**: Adds stopasgroup to the supervisor scripts
- Using Exception.args to serialize/deserialize exceptions
- **Timezones**: Correct calculation of application current time with timezone
- **Remote Debugger**: Set the SO_REUSEADDR option on the socket
- **Django**: Celery ignores exceptions raised during django.setup()
- Use heartbeat setting from application configuration for Broker connection
- **Celery Beat**: Fixed exception caused by next_transit receiving an unexpected argument.
- **Task** Introduce exponential backoff with Task auto-retry
- **AsyncResult**: Remove weak-references to bound methods in AsyncResult promises.
- **Development/Testing**: Allow eager application of canvas structures
- **Command Line**: Flush stderr before exiting with error code 1.
- **Task**: Escapes single quotes in kwargsrepr strings.
- **AsyncResult**: Restore ability to join over ResultSet after fixing celery/3818.
- **Redis Results Backend**: Unsubscribe on message success.
Previously Celery would leak channels, filling the memory of the Redis instance.
- **Task**: Only convert eta to isoformat when it is not already a string.
- **Redis Results Backend**: The result_backend setting now supports rediss:// URIs
- **Canvas** Keyword arguments are passed to tasks in chain as expected.
- **Django** Fix a regression casuing Celery to crash when using Django.
- **Canvas** Chain with one task now runs as expected.
- **Kombu** Celery 4.2 now requires Kombu 4.2 or better.
- GreenletExit is not in __all__ in greenlet.py which can not be imported by Python 3.6.
The import was adjusted to work on Python 3.6 as well.
- Fixed a regression that occured during the development of Celery 4.2 which caused celery report to crash when Django is installed.
- Matched the behavior of GroupResult.as_tuple() to that of AsyncResult.as_tuple().
The group's parent is now serialized correctly.
- Use Redis coercion mechanism for converting URI query parameters.
- Fixed the representation of GroupResult.
- Configuration: CELERY_SEND_EVENTS instead of CELERYD_SEND_EVENTS for 3.1.x compatibility
- App: Restore behavior so Broadcast queues work.
- Sphinx: Make appstr use standard format
- App: Make id, name always accessible from logging.Formatter via extra
- Worker: Add worker_shutting_down signal
- PyPy: Support PyPy version 5.8.0
- Results: Elasticsearch: Fix serializing keys
- Canvas: Deserialize all tasks in a chain
- Systemd: Recover loglevel for ExecStart in systemd config
- Sphinx: Use the Sphinx add_directive_to_domain API.
- App: Pass properties to before_task_publish signal
- Results: Add SSL option for Redis backends
- Beat: celery.schedule.crontab: fix reduce
- State: Fix celery issues when using flower REST API
- Results: Elasticsearch: Fix serializing document id.
- Beat: Make shallow copy of schedules dictionary
- Beat: Populate heap when periodic tasks are changed
- Task: Allow class methods to define tasks
- Platforms: Always return boolean value when checking if signal is supported.
- Canvas: Avoid duplicating chains in chords
- Canvas: Lookup task only if list has items
- Results: Allow unicode message for exception raised in task
- Python3: Support for Python 3.6
- App: Fix retried tasks with expirations
- * Fixes items format route in docs
- Utils: Fix maybe_make_aware
- Task: Fix task ETA issues when timezone is defined in configuration
- Concurrency: Consumer does not shutdown properly when embedded in gevent application
- Canvas: Fix 3725: Task replaced with group does not complete
- Task: Correct order in chains with replaced tasks
- Result: Enable synchronous execution of sub-tasks
- Task: Fix request context for blocking task apply (added hostname)
- Utils: Fix task argument handling
- Beat: Provide a transparent method to update the Scheduler heap
- Beat: Specify default value for pidfile option of celery beat.
- Results: Elasticsearch: Stop generating a new field every time when a new result is being put
- Results: Elasticsearch now reuses fields when new results are added.
- Results: Fixed MongoDB integration when using binary encodings
- Worker: Making missing ``*args`` and ``kwargs`` in Task protocol 1
return empty value in protocol 2.
- App: Fixed :exc:`TypeError` in AMQP when using deprecated signal
- Beat: Added a transparent method to update the scheduler heap.
- Task: Fixed handling of tasks with keyword arguments on Python 3
- Task: Fixed request context for blocking task apply by adding missing
hostname attribute.
- Task: Added option to run subtasks synchronously with
``disable_sync_subtasks`` argument.
- App: Fixed chaining of replaced tasks.
- Canvas: Fixed bug where replaced tasks with groups were not completing
- Worker: Fixed problem where consumer does not shutdown properly when
embedded in a gevent application.
- Results: Added support for using AWS DynamoDB as a result backend.
- Testing: Added caching on pip installs.
- Worker: Prevent consuming queue before ready on startup.
- App: Fixed task ETA issues when timezone is defined in configuration
- Utils: ``maybe_make_aware`` should not modify datetime when it is
already timezone-aware.
- App: Fixed retrying tasks with expirations.
- Results: Allow unicode message for exceptions raised in task
- Canvas: Fixed :exc:`IndexError` raised when chord has an empty header.
- Canvas: Avoid duplicating chains in chords.
- Utils: Allow class methods to define tasks.
- Beat: Populate heap when periodic tasks are changed.
- Results: Added support for Elasticsearch backend options settings.
- Events: Ensure ``Task.as_dict()`` works when not all information about
task is available.
- Schedules: Fixed pickled crontab schedules to restore properly.
- Results: Added SSL option for redis backends
* Requirements: Now depends on Kombu 4.0.2.
* Tasks: Fixed problem with JSON serialization of group
* Worker: Fixed JSON serialization issue when using inspect active and friends
* App: Fixed saferef errors when using signals
* Prefork: Fixed bug with pack requiring bytes argument on Python 2.7.5 and earlier
* Tasks: Saferepr did not handle unicode in bytestrings on Python 2
* Testing: Added new celery_worker_paremeters fixture.
* Tasks: Added new app argument to GroupResult.restore
This makes the restore method behave the same way as the GroupResult constructor.
* Tasks: Fixed type checking crash when task takes *args on Python 3
* Documentation and examples improvements
======
- **Worker**: 3.1.15 broke ``-Ofair`` behavior.
This regression could result in all tasks executing
in a single child process if ``-Ofair`` was enabled.
- **Canvas**: ``celery.signature`` now properly forwards app argument
in all cases.
- **Task**: ``.retry()`` did not raise the exception correctly
when called without a current exception.
- **Worker**: The ``enable_events`` remote control command
disabled worker-related events by mistake.
- **Django**: Adds support for Django 1.7 class names in INSTALLED_APPS
when using ``app.autodiscover_tasks()``.
- **Sphinx**: ``celery.contrib.sphinx`` now uses ``getfullargspec``
on Python 3.
3.1.15
======
- **Django**: Now makes sure ``django.setup()`` is called
before importing any task modules (Django 1.7 compatibility)
- **Results**: ``result.get()`` was misbehaving by calling
``backend.get_task_meta`` in a finally call leading to
AMQP result backend queues not being properly cleaned up.
3.1.14
======
- **Requirements**
- **Init scripts**: The generic worker init scripts ``status`` command
now gets an accurate pidfile list.
- **Init scripts**: The generic beat script now implements the ``status``
command.
- **Commands**: Multi now writes informational output to stdout instead of
stderr.
- **Worker**: Now ignores not implemented error for ``pool.restart``
- **Task**: Retry no longer raises retry exception when executed in eager mode
- **AMQP Result backend**: Now ensured ``on_interval`` is called at least
every second for blocking calls to properly propagate parent errors.
- **Django**: Compatibility with Django 1.7 on Windows.
- **Programs**: `--umask` argument can be now specified in both octal (if
starting
======
Security Fixes
--------------
* [Security: `CELERYSA-0002`_] Insecure default umask.
The built-in utility used to daemonize the Celery worker service sets
an insecure umask by default (umask 0).
This means that any files or directories created by the worker will
end up having world-writable permissions.
Special thanks to Red Hat for originally discovering and reporting the
issue!
This version will no longer set a default umask by default, so if unset
the umask of the parent process will be used.
.. _`CELERYSA-0002`:
http://github.com/celery/celery/tree/master/docs/sec/CELERYSA-0002.txt
News
----
- **Requirements**
- Now depends on :ref:`Kombu 3.0.21 <kombu:version-3.0.21>`.
- Now depends on :mod:`billiard` 3.3.0.18.
- **App**: ``backend`` argument now also sets the :setting:`CELERY_RESULT_BACKEND`
setting.
- **Task**: ``signature_from_request`` now propagates ``reply_to`` so that
the RPC backend works with retried tasks
- **Task**: ``retry`` will no longer attempt to requeue the task if sending
the retry message fails.
Unrelated exceptions being raised could cause a message loop, so it was
better to remove this behavior.
- **Beat**: Accounts for standard 1ms drift by always waking up 0.010s
earlier.
This will adjust the latency so that the periodic tasks will not move
1ms after every invocation.
- Documentation fixes
- **Worker**: Removed an outdated assert statement that could lead to errors
being masked
Celery is an asynchronous task queue/job queue based on distributed message
passing. It is focused on real-time operation, but supports scheduling as well.
The execution units, called tasks, are executed concurrently on a single or more
worker servers using multiprocessing, Eventlet, or gevent. Tasks can execute
asynchronously (in the background) or synchronously (wait until ready).
