Debugging Celery

Related to [[debugging-python]].

Debugging dockerized celery services is not as straightforward as debugging simple python services. One issue is that the normal breakpoint() doesn't work for celery. The other issue is that celery doesn't come with a auto-reload feature, the same way frameworks like flask do.

So, we need to do a couple of things to make debugging celery docker services easier:

• use celery's own remote debugger.
• open up the ports the debugger generally listens to
• use an external hot reloading tool

Changes to docker-compose.yml

stdin_open: true
tty: true
ports:
	- 6900-7000:6900-7000
environment:
	- CELERY_RDB_HOST=0.0.0.0. # to be able to telnet from outside the container
	- PYTHONUNBUFFERED=1       # any non-empty string will do

Hot reloading celery

pip install watchdog

Instead of running celery directly like celery worker --app=..., run it indirectly using watchmedo:

watchmedo auto-restart --directory=./ --pattern=*.py --recursive -- celery \
worker --app=...

More here.

Add breakpoints in code

Add a breakpoint using:

from celery.contrib import rdb

# SOME PYTHON CODE
rdb.set_trace()
# SOME MORE PYTHON CODE

More here.

Running containers

Unfortunately, celery's rdb doesn't work with docker-compose up. There seems to be an issue with the way service ports are exposed via up. More on that here.

Run the service using docker-compose run.

docker-compose run --service-ports celery

Using the debugger

Invoke breakpoint. On invocation, remote debugger will take over: Remote debugger:<PORT>: Ready to connect: telnet 0.0.0.0 <PORT>. It's important that the debugger is listening on 0.0.0.0 as it will enable us to connect to it from outside the container.

telnet localhost <PORT>