Celery, Periodic Task Execution, With Concurrency

I would like to launch a periodic task every second but only if the previous task ended (db polling to send task to celery). In the Celery documentation they are using the Django

Solution 1:

From the Django Cache Framework documentation about local-memory cache:

Note that each process will have its own private cache instance, which means no cross-process caching is possible.

So basically your workers are each dealing with their own cache. If you need a low resource cost cache backend I would recommend File Based Cache or Database Cache, both allow cross-process.