What's Changed

• make the tests run on python 3.13 for gcs backend by @auvipy in #9677
• Added DeepWiki to README by @Nusnus in #9683
• Limit redis to <=v5.2.1 to match Kombu by @Nusnus in #9693
• Use EX_OK instead of literal zero by @Nusnus in #9684
• Make wheel metadata reproducible by @cjwatson in #9687
• let celery install from kombu dependencies for better align by @auvipy in #9696
• Fix stamping documentation to clarify stamped_headers key is optional in visitor methods by @Nusnus in #9697
• Support apply_async without queue argument on quorum queues by @bityob in #9686
• Updated rabbitmq doc about using quorum queues with task routes by @bityob in #9707
• Add: Dumper Unit Test by @yiyu0x in #9711
• Add unit test for event.group_from by @rogerforlife in #9709
• refactor: add beat_cron_starting_deadline documentation warning by @alirafiei75 in #9712
• fix: resolve issue #9569 by supporting distinct broker transport options for workers by @yiyu0x in #9695
• Fixes issue with retry callback arguments in DelayedDelivery by @maccinza in #9708
• get_exchange-unit-test by @rogerforlife in #9710
• ISSUE-9704: Update documentation of result_expires, filesystem backend is supported by @MattiasDC in #9716
• update to blacksmith ubuntu 24.04 by @auvipy in #9717
• Added unit tests for celery.utils.iso8601 by @jaiganeshs21 in #9725
• Update introduction.rst docs by @auvipy in #9728
• Prepare for release: v5.5.3 by @Nusnus in #9732

New Contributors

• @bityob made their first contribution in #9686
• @yiyu0x made their first contribution in #9711
• @rogerforlife made their first contribution in #9709
• @alirafiei75 made their first contribution in #9712
• @maccinza made their first contribution in #9708
• @MattiasDC made their first contribution in #9716
• @jaiganeshs21 made their first contribution in #9725

Full Changelog: v5.5.2...v5.5.3

What's Changed

• Fix calculating remaining time across DST changes by @imrivera in #9669
• Remove setup_logger from COMPAT_MODULES by @rconradharris in #9668
• Fix mongodb bullet and fix github links in contributions section by @jabberwock404 in #9672
• Prepare for release: v5.5.2 by @Nusnus in #9675

New Contributors

• @imrivera made their first contribution in #9669
• @rconradharris made their first contribution in #9668
• @jabberwock404 made their first contribution in #9672

Full Changelog: v5.5.1...v5.5.2

What's Changed

• Fixed "AttributeError: list object has no attribute strip" with quorum queues and failover brokers by @Nusnus in #9657
• Prepare for release: v5.5.1 by @Nusnus in #9660

Full Changelog: v5.5.0...v5.5.1

Celery v5.5.0 is now available.

Key Highlights

See What's new in Celery 5.5 for a complete overview or read the main highlights below.

Redis Broker Stability Improvements

Long-standing disconnection issues with the Redis broker have been identified and resolved in Kombu 5.5.0, which is included with this release. These improvements significantly enhance stability when using Redis as a broker.

Additionally, the Redis backend now has better exception handling with the new exception_safe_to_retry feature, which improves resilience during temporary Redis connection issues. See Redis backend settings for complete documentation.

Contributed by @drienkop in #9614.

pycurl replaced with urllib3

Replaced the pycurl dependency with urllib3.

We're monitoring the performance impact of this change and welcome feedback from users who notice any significant differences in their environments.

Contributed by @spawn-guy in Kombu #2134 and integrated in Celery via #9526.

RabbitMQ Quorum Queues Support

Added support for RabbitMQ's new Quorum Queues feature, including compatibility with ETA tasks. This implementation has some limitations compared to classic queues, so please refer to the documentation for details.

Native Delayed Delivery is automatically enabled when quorum queues are detected to implement the ETA mechanism.

See Using Quorum Queues for complete documentation.

Configuration options:

• broker_native_delayed_delivery_queue_type: Specifies the queue type for delayed delivery (default: quorum)
• task_default_queue_type: Sets the default queue type for tasks (default: classic)
• worker_detect_quorum_queues: Controls automatic detection of quorum queues (default: True)

Contributed in #9207, #9121, and #9599.

For details regarding the 404 errors, see New Year's Security Incident.

Soft Shutdown Mechanism

Soft shutdown is a time limited warm shutdown, initiated just before the cold shutdown. The worker will allow worker_soft_shutdown_timeout seconds for all currently executing tasks to finish before it terminates. If the time limit is reached, the worker will initiate a cold shutdown and cancel all currently executing tasks.

This feature is particularly valuable when using brokers with visibility timeout mechanisms, such as Redis or SQS. It allows the worker enough time to re-queue tasks that were not completed before exiting, preventing task loss during worker shutdown.

See Stopping the worker for complete documentation on worker shutdown types.

Configuration options:

• worker_soft_shutdown_timeout: Sets the duration in seconds for the soft shutdown period (default: 0.0, disabled)
• worker_enable_soft_shutdown_on_idle: Controls whether soft shutdown should be enabled even when the worker is idle (default: False)

Contributed by @Nusnus in #9213, #9231, and #9238.

Pydantic Support

New native support for Pydantic models in tasks. This integration allows you to leverage Pydantic's powerful data validation and serialization capabilities directly in your Celery tasks.

Example usage:

from pydantic import BaseModel
from celery import Celery

app = Celery('tasks')

class ArgModel(BaseModel):
    value: int

class ReturnModel(BaseModel):
    value: str

@app.task(pydantic=True)
def x(arg: ArgModel) -> ReturnModel:
    # args/kwargs type hinted as Pydantic model will be converted
    assert isinstance(arg, ArgModel)

    # The returned model will be converted to a dict automatically
    return ReturnModel(value=f"example: {arg.value}")

See Argument validation with Pydantic for complete documentation.

Configuration options:

• pydantic=True: Enables Pydantic integration for the task
• pydantic_strict=True/False: Controls whether strict validation is enabled (default: False)
• pydantic_context={...}: Provides additional context for validation
• pydantic_dump_kwargs={...}: Customizes serialization behavior

Contributed by @mathiasertl in #9023, #9319, and #9393.

Google Pub/Sub Transport

New support for Google Cloud Pub/Sub as a message transport, expanding Celery's cloud integration options.

See Google Pub/Sub Transport for complete documentation.

For the Google Pub/Sub support you have to install additional dependencies:

$ pip install "celery[gcpubsub]"

Then configure your Celery application to use the Google Pub/Sub transport:

broker_url = 'gcpubsub://projects/project-id'

Contributed by @haimjether in #9351.

Python 3.13 Support

Official support for Python 3.13. All core dependencies have been updated to ensure compatibility, including Kombu and py-amqp.

This release maintains compatibility with Python 3.8 through 3.13, as well as PyPy 3.10+.

Contributed by @Nusnus in #9309 and #9350.

REMAP_SIGTERM Support

The "REMAP_SIGTERM" feature, previously undocumented, has been tested, documented, and is now officially supported. This feature allows you to remap the SIGTERM signal to SIGQUIT, enabling you to initiate a soft or cold shutdown using TERM instead of QUIT.

This is particularly useful in containerized environments where SIGTERM is the standard signal for graceful termination.

See Cold Shutdown documentation for more info.

To enable this feature, set the environment variable:

export REMAP_SIGTERM="SIGQUIT"

Contributed by @Nusnus in #9461.

Database Backend Improvements

New create_tables_at_setup option for the database backend. This option controls when database tables are created, allowing for non-lazy table creation.

By default (create_tables_at_setup=True), tables are created during backend initialization. Setting this to False defers table creation until they are actually needed, which can be useful in certain deployment scenarios where you want more control over database schema management.

See Database backend settings for complete documentation.

Configuration:

app.conf.result_backend = 'db+sqlite:///results.db'
app.conf.database_create_tables_at_setup = False

Contributed by @MarcBresson in #9228.

What's Changed

• (docs): use correct version celery v.5.4.x by @FraCata00 in #8975
• Update mypy to 1.10.0 by @pyup-bot in #8977
• Limit pymongo<4.7 when Python <= 3.10 due to breaking changes in 4.7 by @Nusnus in #8988
• Bump pytest from 8.1.1 to 8.2.0 by @dependabot in #8987
• Update README to Include FastAPI in Framework Integration Section by @pedroimpulcetto in #8978
• Clarify return values of ..._on_commit methods by @browniebroke in #8984
• add kafka broker docs by @thuibr in #8935
• Limit pymongo<4.7 regardless of Python version by @Nusnus in #8999
• Update pymongo[srv] requirement from <4.7,>=4.0.2 to >=4.0.2,<4.8 by @dependabot in #9000
• Update elasticsearch requirement from <=8.13.0 to <=8.13.1 by @dependabot in #9004
• security: SecureSerializer: support generic low-level serializers by @shirsa in #8982
• don't kill if pid same as file (#8997) by @lewijw in #8998
• Update cryptography to 42.0.6 by @pyup-bot in #9005
• Bump cryptography from 42.0.6 to 42.0.7 by @dependabot in #9009
• don't kill if pid same as file (#8997) (#8998) by @lewijw in #9007
• Added -vv to unit, integration and smoke tests by @Nusnus in #9014
• SecuritySerializer: ensure pack separator will not be conflicted with serialized fields by @shirsa in #9010
• Update sphinx-click to 5.2.2 by @pyup-bot in #9025
• Bump sphinx-click from 5.2.2 to 6.0.0 by @dependabot in ...

Celery v5.5.0 Release Candidate 5 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

See What’s new in Celery 5.5 (Immunity) or read the main highlights below.

Using Kombu 5.5.0rc3

The minimum required Kombu version has been bumped to 5.5.0.
Kombu is currently at 5.5.0rc3.

Complete Quorum Queues Support

A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues.

After upgrading to this version, please share your feedback on the quorum queues support.

Relevant Issues: #9207, #6067

• New documentation.
• New broker_native_delayed_delivery_queue_type configuration option.

New support for Google Pub/Sub transport

After upgrading to this version, please share your feedback on the Google Pub/Sub transport support.

Relevant Issues: #9351

Python 3.13 Improved Support

Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp.

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

• New documentation for each shutdown type.
• New worker_soft_shutdown_timeout configuration option.
• New worker_enable_soft_shutdown_on_idle configuration option.

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• Bump mypy from 1.13.0 to 1.14.0 by @dependabot in #9476
• Fix cassandra backend port settings not working by @kairi003 in #9465
• Unroll group when a group with a single item is chained using the | operator by @thedrow in #9456
• fix(django): catch the right error when trying to close db connection by @Lotram in #9392
• Replacing a task with a chain which contains a group now returns a result instead of hanging by @thedrow in #9484
• Avoid using a group of one as it is now unrolled into a chain by @thedrow in #9510
• Link to the correct IRC network by @yigitsever in #9509
• Bump pytest-github-actions-annotate-failures from 0.2.0 to 0.3.0 by @dependabot in #9504
• Update canvas.rst to fix output result from chain object by @kamalfarahani in #9502
• Unauthorized Changes Cleanup by @Nusnus in #9528
• [RE-APPROVED] fix(django): catch the right error when trying to close db connection by @Nusnus in #9529
• [RE-APPROVED] Link to the correct IRC network by @Nusnus in #9531
• [RE-APPROVED] Update canvas.rst to fix output result from chain object by @Nusnus in #9532
• Update test-ci-base.txt by @auvipy in #9539
• Update install-pyenv.sh by @auvipy in #9540
• Update elasticsearch requirement from <=8.17.0 to <=8.17.1 by @dependabot in #9518
• Bump google-cloud-firestore from 2.19.0 to 2.20.0 by @dependabot in #9493
• Bump mypy from 1.14.0 to 1.14.1 by @dependabot in #9483
• Update elastic-transport requirement from <=8.15.1 to <=8.17.0 by @dependabot in #9490
• Update Dockerfile by adding missing Python version 3.13 by @auvipy in #9549
• Fix typo for default of sig by @daveisfera in #9495
• fix(crontab): resolve constructor type conflicts by @pcrock-thmdo in #9551
• worker_max_memory_per_child: kilobyte is 1024 bytes by @mksm in #9553
• Fix formatting in quorum queue docs by @HenrikOssipoff in #9555
• Bump cryptography from 44.0.0 to 44.0.1 by @dependabot in #9556
• Fix the send_task method when detecting if the native delayed delivery approach is available by @mikhaillazko in #9552
• Reverted PR #7814 & minor code improvement by @MehrazRumman in #9494
• Improved donation and sponsorship visibility by @Nusnus in #9558
• Updated the Getting Help section, replacing deprecated with new resources by @Nusnus in #9559
• Fixed django example by @Nusnus in #9562
• Bump Kombu to v5.5.0rc3 by @Nusnus in #9564
• Bump ephem from 4.1.6 to 4.2 by @dependabot in #9565
• Bump pytest-celery to v1.2.0 by @Nusnus in #9568
• Remove dependency on pycurl by @jmsmkn in #9526
• Set TestWorkController.test by @WilliamDEdwards in #9574
• Fixed bug when revoking by stamped headers a stamp that does not exist by @Nusnus in #9575
• Canvas Stamping Doc Fixes by @Nusnus in #9578
• Bugfix: Chord with a chord in header doesn't invoke error callback on inner chord header failure (default config) by @Nusnus in #9580
• Prepare for (pre) release: v5.5.0rc5 by @Nusnus in #9582

New Contributors

• @kairi003 made their first contribution in ...

Celery v5.5.0 Release Candidate 4 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

See What’s new in Celery 5.5 (Immunity) or read the main highlights below.

Using Kombu 5.5.0rc2

The minimum required Kombu version has been bumped to 5.5.0.
Kombu is current at 5.5.0rc2.

Complete Quorum Queues Support

A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues.

After upgrading to this version, please share your feedback on the quorum queues support.

Relevant Issues: #9207, #6067

• New documentation.
• New broker_native_delayed_delivery_queue_type configuration option.

New support for Google Pub/Sub transport

After upgrading to this version, please share your feedback on the Google Pub/Sub transport support.

Relevant Issues: #9351

Python 3.13 Improved Support

Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp.

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

• New documentation for each shutdown type.
• New worker_soft_shutdown_timeout configuration option.
• New worker_enable_soft_shutdown_on_idle configuration option.

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• Bugfix: SIGQUIT not initiating cold shutdown when task_acks_late=False by @Nusnus in #9461
• Fixed pycurl dep with Python 3.8 by @Nusnus in #9471
• Update elasticsearch requirement from <=8.16.0 to <=8.17.0 by @dependabot in #9469
• Bump pytest-subtests from 0.13.1 to 0.14.1 by @dependabot in #9459
• documentation: Added a type annotation to the periodic task example by @Avamander in #9473
• Prepare for (pre) release: v5.5.0rc4 by @Nusnus in #9474

New Contributors

• @Avamander made their first contribution in #9473

Full Changelog: v5.5.0rc3...v5.5.0rc4

Celery v5.5.0 Release Candidate 3 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

See What’s new in Celery 5.5 (Immunity) or read the main highlights below.

Using Kombu 5.5.0rc2

The minimum required Kombu version has been bumped to 5.5.0.
Kombu is current at 5.5.0rc2.

Complete Quorum Queues Support

A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues.

After upgrading to this version, please share your feedback on the quorum queues support.

Relevant Issues: #9207, #6067

• New documentation.
• New broker_native_delayed_delivery_queue_type configuration option.

New support for Google Pub/Sub transport

After upgrading to this version, please share your feedback on the Google Pub/Sub transport support.

Relevant Issues: #9351

Python 3.13 Improved Support

Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp.

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

• New documentation for each shutdown type.
• New worker_soft_shutdown_timeout configuration option.
• New worker_enable_soft_shutdown_on_idle configuration option.

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• Document usage of broker_native_delayed_delivery_queue_type by @thedrow in #9419
• Adjust section in what's new document regarding quorum queues support by @thedrow in #9420
• Update pytest-rerunfailures to 15.0 by @pyup-bot in #9422
• Document group unrolling by @thedrow in #9421
• fix small typo acces -> access by @sharuzzaman in #9434
• Update cryptography to 44.0.0 by @pyup-bot in #9437
• Added pypy to Dockerfile by @Nusnus in #9438
• Skipped flaky tests on pypy (all pass after ~10 reruns) by @Nusnus in #9439
• Allowing managed credentials for azureblockblob by @PieterBlomme in #9430
• Allow passing Celery objects to the Click entry point by @0x2b3bfa0 in #9426
• support Request termination for gevent by @woutdenolf in #9440
• Prevent event_mask from being overwritten. by @Androidown in #9432
• Update pytest to 8.3.4 by @pyup-bot in #9444
• Prepare for (pre) release: v5.5.0rc3 by @Nusnus in #9450

New Contributors

• @sharuzzaman made their first contribution in #9434
• @PieterBlomme made their first contribution in #9430

Full Changelog: v5.5.0rc2...v5.5.0rc3

Celery v5.5.0 Release Candidate 2 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

See What’s new in Celery 5.5 (Immunity) or read the main highlights below.

Using Kombu 5.5.0rc2

The minimum required Kombu version has been bumped to 5.5.0.
Kombu is current at 5.5.0rc2.

Complete Quorum Queues Support

A completely new ETA mechanism was developed to allow full support with RabbitMQ Quorum Queues.

After upgrading to this version, please share your feedback on the quorum queues support.

Relevant Issues: #9207, #6067

• New documentation.
• New broker_native_delayed_delivery_queue_type configuration option.

New support for Google Pub/Sub transport

After upgrading to this version, please share your feedback on the Google Pub/Sub transport support.

Relevant Issues: #9351

Python 3.13 Improved Support

Additional dependencies have been migrated successfully to Python 3.13, including Kombu and py-amqp.

Previous Pre-release Highlights

Python 3.13 Initial Support

This release introduces the initial support for Python 3.13 with Celery.

After upgrading to this version, please share your feedback on the Python 3.13 support.

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

• New documentation for each shutdown type.
• New worker_soft_shutdown_timeout configuration option.
• New worker_enable_soft_shutdown_on_idle configuration option.

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• Fix: Treat dbm.error as a corrupted schedule file by @stumpylog in #9331
• Pin pre-commit to latest version 4.0.1 by @pyup-bot in #9343
• Added Python 3.13 to Dockerfiles by @Nusnus in #9350
• Skip test_pool_restart_import_modules on PyPy due to test issue by @Nusnus in #9352
• Update elastic-transport requirement from <=8.15.0 to <=8.15.1 by @dependabot in #9347
• added dragonfly logo by @auvipy in #9353
• Update README.rst by @auvipy in #9354
• Update README.rst by @auvipy in #9355
• Update mypy to 1.12.0 by @pyup-bot in #9356
• Bump Kombu to v5.5.0rc1 by @Nusnus in #9357
• Fix celery --loader option parsing by @0x2b3bfa0 in #9361
• Add support for Google Pub/Sub transport by @haimjether in #9351
• Add native incr support for GCSBackend by @haimjether in #9302
• fix(perform_pending_operations): prevent task duplication on shutdown… by @moaddib666 in #9348
• Update grpcio to 1.67.0 by @pyup-bot in #9365
• Update google-cloud-firestore to 2.19.0 by @pyup-bot in #9364
• Annotate celery/utils/timer2.py by @hmnfalahi in #9362
• Update cryptography to 43.0.3 by @pyup-bot in #9366
• Update mypy to 1.12.1 by @pyup-bot in #9368
• Bump mypy from 1.12.1 to 1.13.0 by @dependabot in #9373
• Pass timeout and confirm_timeout to producer.publish() by @thedrow in #9374
• Bump Kombu to v5.5.0rc2 by @Nusnus in #9382
• Bump pytest-cov from 5.0.0 to 6.0.0 by @dependabot in #9388
• default strict to False for pydantic tasks by @mathiasertl in #9393
• Only log that global QoS is disabled if using amqp by @thedrow in #9395
• chore: update sponsorship logo by @Niennienzz in #9398
• Allow custom hostname for celery_worker in celery.contrib.pytest / celery.contrib.testing.worker by @SlowMo24 in #9405
• Removed docker-docs from CI (optional job, malfunctioning) by @Nusnus in #9406
• Added a utility to format changelogs from the auto-generated GitHub release notes by @Nusnus in #9408
• Bump codecov/codecov-action from 4 to 5 by @dependabot in #9412
• Update elasticsearch requirement from <=8.15.1 to <=8.16.0 by @dependabot in #9410
• Native Delayed Delivery in RabbitMQ by @thedrow in #9207
• Prepare for (pre) release: v5.5.0rc2 by @Nusnus in #9416

New Contributors

• @0x2b3bfa0 made their first contribution in #9361
• @hmnfalahi made their first contribution in #9362
• @Niennienzz made their first contribution in #9398
• @SlowMo24 made their first contribution in #9405

Full Changelog: v5.5.0rc1...v5.5.0rc2

Celery v5.5.0 Release Candidate 1 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

See What’s new in Celery 5.5 (Immunity) or read the main highlights below.

Python 3.13 Initial Support

This release introduces the initial support for Python 3.13 with Celery.

After upgrading to this version, please share your feedback on the Python 3.13 support.

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

• New documentation for each shutdown type.
• New worker_soft_shutdown_timeout configuration option.
• New worker_enable_soft_shutdown_on_idle configuration option.

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• Added Blacksmith.sh to the Sponsors section in the README by @Nusnus in #9323
• Revert "Added Blacksmith.sh to the Sponsors section in the README" by @Nusnus in #9324
• Added Blacksmith.sh to the Sponsors section in the README by @Nusnus in #9325
• Added missing " |oc-sponsor-3|” in README by @Nusnus in #9326
• Use Blacksmith SVG logo by @Nusnus in #9327
• Updated Blacksmith SVG logo by @Nusnus in #9328
• Revert "Updated Blacksmith SVG logo" by @Nusnus in #9329
• Update pymongo to 4.10.0 by @pyup-bot in #9330
• Update pymongo to 4.10.1 by @pyup-bot in #9332
• Update user guide to recommend delay_on_commit by @browniebroke in #9333
• Pin pre-commit to latest version 4.0.0 (Python 3.9+) by @pyup-bot in #9334
• Update ephem to 4.1.6 by @pyup-bot in #9336
• Updated Blacksmith SVG logo by @Nusnus in #9337
• Prepare for (pre) release: v5.5.0rc1 by @Nusnus in #9341

Full Changelog: v5.5.0b4...v5.5.0rc1

Celery v5.5.0 Beta 4 is now available for testing.
Please help us test this version and report any issues.

Key Highlights

Python 3.13 Initial Support

This release introduces the initial support for Python 3.13 with Celery.

After upgrading to this version, please share your feedback on the Python 3.13 support.

Previous Pre-release Highlights

Soft Shutdown

The soft shutdown is a new mechanism in Celery that sits between the warm shutdown and the cold shutdown. It sets a time limited "warm shutdown" period, during which the worker will continue to process tasks that are already running. After the soft shutdown ends, the worker will initiate a graceful cold shutdown, stopping all tasks and exiting.

The soft shutdown is disabled by default, and can be enabled by setting the new configuration option worker_soft_shutdown_timeout. If a worker is not running any task when the soft shutdown initiates, it will skip the warm shutdown period and proceed directly to the cold shutdown unless the new configuration option worker_enable_soft_shutdown_on_idle is set to True. This is useful for workers that are idle, waiting on ETA tasks to be executed that still want to enable the soft shutdown anyways.

The soft shutdown can replace the cold shutdown when using a broker with a visibility timeout mechanism, like Redis or SQS, to enable a more graceful cold shutdown procedure, allowing the worker enough time to re-queue tasks that were not completed (e.g., Restoring 1 unacknowledged message(s)) by resetting the visibility timeout of the unacknowledged messages just before the worker exits completely.

After upgrading to this version, please share your feedback on the new Soft Shutdown mechanism.

Relevant Issues: #9213, #9231, #9238

• New documentation for each shutdown type.
• New worker_soft_shutdown_timeout configuration option.
• New worker_enable_soft_shutdown_on_idle configuration option.

REMAP_SIGTERM

The REMAP_SIGTERM "hidden feature" has been tested, documented and is now officially supported.
This feature allows users to remap the SIGTERM signal to SIGQUIT, to initiate a soft or a cold shutdown using TERM
instead of QUIT.

Pydantic Support

This release introduces support for Pydantic models in Celery tasks.
For more info, see the new pydantic example and PR #9023 by @mathiasertl.

After upgrading to this version, please share your feedback on the new Pydantic support.

Redis Broker Stability Improvements

The root cause of the Redis broker instability issue has been identified and resolved in the v5.4.0 release of Kombu, which should resolve the disconnections bug and offer additional improvements.

After upgrading to this version, please share your feedback on the Redis broker stability.

Relevant Issues: #7276, #8091, #8030, #8384

Quorum Queues Initial Support

This release introduces the initial support for Quorum Queues with Celery.

See new configuration options for more details:

• task_default_queue_type
• worker_detect_quorum_queues

After upgrading to this version, please share your feedback on the Quorum Queues support.

Relevant Issues: #6067, #9121

What's Changed

• Correct the error description in exception message when validate soft_time_limit by @narasux in #9246
• Update msgpack to 1.1.0 by @pyup-bot in #9249
• chore(utils/time.py): rename _is_ambigious -> _is_ambiguous by @pachewise in #9248
• Reduced Smoke Tests to min/max supported python (3.8/3.12) by @Nusnus in #9252
• Update pytest to 8.3.3 by @pyup-bot in #9253
• Update elasticsearch requirement from <=8.15.0 to <=8.15.1 by @dependabot in #9255
• update mongodb without deprecated [srv] extra requirement by @fmigneault in #9258
• blacksmith.sh: Migrate workflows to Blacksmith by @blacksmith-sh in #9261
• Fixes #9119: inject dispatch_uid for retry-wrapped receivers by @pachewise in #9247
• Run all smoke tests CI jobs together by @Nusnus in #9263
• Improve documentation on visibility timeout by @kylez-ithaka in #9264
• Bump pytest-celery to 1.1.2 by @Nusnus in #9267
• Added missing "app.conf.visibility_timeout" in smoke tests by @Nusnus in #9266
• Improved stability with t/smoke/tests/test_consumer.py by @Nusnus in #9268
• Improved Redis container stability in the smoke tests by @Nusnus in #9271
• Disabled EXHAUST_MEMORY tests in Smoke-tasks by @Nusnus in #9272
• Marked xfail for test_reducing_prefetch_count with Redis - flaky test by @Nusnus in #9273
• Fixed pypy unit tests random failures in the CI by @Nusnus in #9275
• Fixed more pypy unit tests random failures in the CI by @Nusnus in #9278
• Fix Redis container from aborting randomly by @Nusnus in #9276
• Run Integration & Smoke CI tests together after unit tests passes by @Nusnus in #9280
• Added "loglevel verbose" to Redis containers in smoke tests by @Nusnus in #9282
• Fixed Redis error in the smoke tests: "Possible SECURITY ATTACK detected" by @Nusnus in #9284
• Refactored the smoke tests github workflow by @Nusnus in #9285
• Increased --reruns 3->4 in smoke tests by @Nusnus in #9286
• Improve stability of smoke tests (CI and Local) by @Nusnus in #9287
• Fixed Smoke tests CI "test-case" lables (specific instead of general) by @Nusnus in #9288
• Use assert_log_exists instead of wait_for_log in worker smoke tests by @Nusnus in #9290
• Optimized t/smoke/tests/test_worker.py by @Nusnus in #9291
• Enable smoke tests dockers check before each test starts by @Nusnus in #9292
• Relaxed smoke tests flaky tests mechanism by @Nusnus in #9293
• Updated quorum queue detection to handle multiple broker instances by @bkienker in #9294
• Non-lazy table creation for database backend by @MarcBresson in #9228
• Pin pymongo to latest version 4.9 by @pyup-bot in #9297
• Bump pymongo from 4.9 to 4.9.1 by @dependabot in #9298
• Bump Kombu to v5.4.2 by @Nusnus in #9304
• Use rabbitmq:3 in stamping smoke tests by @Nusnus in #9307
• Bump pytest-celery to 1.1.3 by @Nusnus in #9308
• Added Python 3.13 Support by @Nusnus in #9309
• Add log when global qos is disabled by @thedrow in #9296
• Added official release docs (whatsnew) for v5.5 by @Nusnus in #9312
• Enable Codespell autofix by @Nusnus in #9313
• Pydantic typehints: Fix optional, allow generics by @mathiasertl in #9319
• Prepare for (pre) release: v5.5.0b4 by @Nusnus in #9322

New Contributors

• @narasux made their first contribution in #9246
• @fmigneault made their first contribution in #9258
• @blacksmith-sh made their first contribution in #9261
• @kylez-ithaka made their first contribution in #9264
• @bkienker made their first contribution in #9294
• @MarcBresson made their first contribution in #9228

Full Changelog: v5.5.0b3...v5.5.0b4