Learnings

<p>Preventing Data Loss When Using Messaging: </p>

<p>Celery with heavy workloads (TL;DR use RabbitMQ as broker and Redis as result backend): </p>

<p>Creating pickleable exceptions: </p>

<p>Celery in production: Three more years of fixing bugs </p>

<p>In celery, when retrying tasks, nothing will run after a <code>raise (...)</code>, even if the are no more retries left.</p>

<p>How to get quantity of tasks in Celery and Redis from the default queue.</p> </br> <code> import redis from django.conf import settings queue_name = "celery" client = redis.Redis.from_url(settings.REDIS_URL) length = client.llen(queue_name) </code>

<p>Celery Once allows you to prevent multiple execution and queuing of celery tasks: </p>

<p>Using Celery on Heroku has been recently updated: </p>

<p>Common Issues Using Celery (And Other Task Queues) </p>

<p>Celery supports exponential backoff options since 4.2 #Task.retry_backoff</p>

<p>Must read article on common Celery issues: </p>

<p>Testing Celery tasks calls within transactions </p>

<p>jobtastic- Celery tasks plus more awesome </p>

<p>You can set a '--max-memory-per-child' to Celery to automatically replace child processes that exceed the limit. Celery will wait tasks to finish before proceeding #max-memory-per-child-setting</p>

<p>Celery is duplicating tasks for Redis broker: </p>

<p>Celery chord on_error callback does not work as documented: </p>

<p>Easy broker-agnostic way to get number of tasks in a Celery queue: </p>

<p>Debugging code called by celery tasks with rdb #debugging-tasks-remotely-using-pdb</p>

<p>Auto reload Celery upon changes for development environments </p>

<p>Celery has a default configuration that removes every customized logs configuration. So if you change your logs root configuration the changes will only work on the django part. To allow this customization you should do this <code>worker_hijack_root_logger = False</code>: #std:setting-worker_hijack_root_logger</p>

<p>heroku's addons for memcached are not explicitly compatible with Celery as result backends, but you can use them with django-celery-results </p> <p>We created a custom cache for Celery and used [Memcached Cloud]( Our settings for this are looking like:</p> <p>``` INSTALLED_APPS = [ # ... 'django_celery_results', # ... ]</p> <p>CACHES['celery'] = { 'BACKEND': 'django_bmemcached.memcached.BMemcached', 'LOCATION': MEMCACHED_SERVERS, 'OPTIONS': { 'username': MEMCACHED_USERNAME, 'password': MEMCACHED_PASSWORD, } }</p> <p>CELERY_RESULT_BACKEND = 'django-cache' CELERY_CACHE_BACKEND = 'celery' ```</p> <p>Obs.: To use Memcached Cloud on Django you have to follow <a href="#django">these instructions</a></p>

<p>To list the current Celery tasks names in Redis queue, use:</p> <p><code>import json import base64 from redis import StrictRedis redis_client = StrictRedis.from_url(BROKER_URL) default_queue = redis_client.lrange('celery', 0, -1) print({json.loads(base64.b64decode(json.loads(message.decode('utf-8'))['body']).decode('utf-8'))['task'] for message in default_queue})</code> For reserved and active, use <code>app.control.inspect()</code>: #inspecting-workers</p>

<p>Dramatiq: alternative to Celery and RQ: </p>

<p>Celery 4 does NOT send error emails: </p>