Related to [[debugging-python]].
Debugging dockerized celery services is not as straightforward as debugging
simple python services. One issue is that the normal
breakpoint() doesn't work
for celery. The other issue is that celery doesn't come with a
feature, the same way frameworks like
So, we need to do a couple of things to make debugging celery docker services easier:
- use celery's own remote debugger.
- open up the ports the debugger generally listens to
- use an external hot reloading tool
Changes to docker-compose.yml
stdin_open: true tty: true ports: - 6900-7000:6900-7000 environment: - CELERY_RDB_HOST=0.0.0.0. # to be able to telnet from outside the container - PYTHONUNBUFFERED=1 # any non-empty string will do
Hot reloading celery
pip install watchdog
Instead of running celery directly like
celery worker --app=..., run it
watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery \ worker --app=...
Add breakpoints in code
Add a breakpoint using:
from celery.contrib import rdb # SOME PYTHON CODE rdb.set_trace() # SOME MORE PYTHON CODE
rdb doesn't work with
docker-compose up. There seems
to be an issue with the way service ports are exposed via
up. More on that
Run the service using
docker-compose run --service-ports celery
Using the debugger
Invoke breakpoint. On invocation, remote debugger will take over:
debugger:<PORT>: Ready to connect: telnet 0.0.0.0 <PORT>. It's important that
the debugger is listening on
0.0.0.0 as it will enable us to connect to it
from outside the container.
telnet localhost <PORT>