Celery Heartbeat Not Working

I have set heartbeat in Celery settings: BROKER_HEARTBEAT = 10 I have also set this configuration value in RabbitMQ config: 'heartbeat' => '10', But somehow heartbeats are sti

Solution 1:

the heartbeat of celery worker is application level heartbeat, not AMQP protocol's heartbeat. Each worker periodically send heartbeat event message to "celeryev" event exchange in BROKER. The heartbeat event is forwarded back to worker such worker can know the health status of BROKER. If number of loss heartbeat exceeding a threshold, the worker can do some reconnect action to BROKER.

For the rest of detail, you may check this page The section: BROKER_FAILOVER_STRATEGY describes the actions you may do for dropping from a BROKER.

Solution 2:

Celery worker support AMQP heartbeat definitely. The configuration item BROKER_HEARTBEAT is used to define the heartbeat interval of AMQP client(celery worker). We can find the description of BROKER_HEARTBEAT here Celery Doc!

The possible causes of heartbeat not work:

1. Use a wrong transport such as 'librabbitmq' As celery doc described, only 'pyamqp' transport support BROKER_HEARTBEAT. We need to check whether if librabbitmq package is installed or we can use 'pyamqp' transport in broker url: 'pyamqp://userid:password@hostname:port/virtual_host' rather than 'amqp://userid:password@hostname:port/virtual_host'

2. No event send to celery worker during three heartbeat interval after boot up Check code here to see how heartbeat works! drain_events will be called during worker boot up, see code here! If there's no event sent to celery worker, connection.heartbeat_check will not be called.

By the way, connection.heartbeat_check is defined here!

Hopes to help someone encounter the heartbeat issue.