Celery and RabbitMQ on ArchLinux

TurnToJPG -->


Install and run rabbitmq server via:

$ yaourt rabbitmq
$ rabbitmq-server

Install celery in python virtual enviroment:

$ workon venv2
(venv2) $ pip install celery

Run Simple Tasks

Following python file, named “tasks.py” defines two tasks:

from celery import Celery

app = Celery('tasks', backend='amqp', broker='amqp://')

def print_hello():
    print 'hello there'

def gen_prime(x):
    multiples = []
    results = []
    for i in xrange(2, x+1):
        if i not in multiples:
            for j in xrange(i*i, x+1, i):
    return results

Run it via:

celery worker -A tasks &

Now in another terminal you can use python interractive window for:

from tasks import print_hello
from tasks import gen_prime
primes = gen_prime(1000)
primes = gen_prime(50000)
# CTRL+C will stop it. 
# Access the background worker
primes = gen_prime.delay(50000)
# by the worker executing now, because we configured the backend for application
print primes.get()

Periodic Tasks

Add following lines into the tasks.py:

from celery.task import periodic_task
from datetime import timedelta

def print_minutes():
    print 'Hello, 1 minute reached'

def every_3_seconds():
    print("Running periodic task!")

Now start the tasks.py via following commands:

$ celery -A tasks worker --loglevel=info --beat

You will see Celery output “Running periodic tasks!” every 3 seconds, while every 1 minutes the “Hello, 1 minute reached” will be printed out.

To Be Thought

How to mirgrate it with our heroku web app ?
The periodic_task is good for fetching pages/datas and generate the result, then store the results into the database.

But the webapp should response to user’s http request, this will be the main task.