Debugging dockerized celery services is not as straightforward as debugging simple python services. One issue is that the normal
breakpoint() doesn't work for celery. The other issue is that celery doesn't come with a
auto-reload feature, the same way frameworks like
So, we need to do a couple of things to make debugging celery docker services easier:
- use celery's own remote debugger.
- open up the ports the debugger generally listens to
- use an external hot reloading tool
Changes to docker-compose.yml¶
stdin_open: true tty: true ports: - 6900-7000:6900-7000 environment: - CELERY_RDB_HOST=0.0.0.0. # to be able to telnet from outside the container - PYTHONUNBUFFERED=1 # any non-empty string will do
Hot reloading celery¶
pip install watchdog
Instead of running celery directly like
celery worker --app=..., run it indirectly using
watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery worker --app=...
Add breakpoints in code¶
Add a breakpoint using:
from celery.contrib import rdb # SOME PYTHON CODE rdb.set_trace() # SOME MORE PYTHON CODE
rdb doesn't work with
docker-compose up. There seems to be an issue with the way service ports are exposed via
up. More on that here.
Run the service using
docker-compose run --service-ports celery
Using the debugger¶
Invoke breakpoint. On invocation, remote debugger will take over:
Remote debugger:<PORT>: Ready to connect: telnet 0.0.0.0 <PORT>. It's important that the debugger is listening on
0.0.0.0 as it will enable us to connect to it from outside the container.
telnet localhost <PORT>