How to Use Celery for Task Queues

Celery is an asynchronous task queue/job queue based on distributed message passing. It is used to handle background tasks that can be processed independently of the main application flow. In this tutorial, you’ll learn how to set up and use Celery to manage task queues in your Python applications.

Prerequisites

• Python 3 installed on your machine.
• Basic knowledge of Python programming and working with virtual environments.
• Redis or RabbitMQ installed as a message broker (we’ll use Redis in this example).

1. Installing Celery

Start by creating a new virtual environment and activating it:

python3 -m venv venv
source venv/bin/activate

Install Celery and Redis client libraries:

pip install celery redis

2. Setting Up Your Celery Application

Next, create a new Python file named tasks.py:

touch tasks.py

Open tasks.py in your text editor and add the following code:

import time
from celery import Celery

app = Celery('tasks', broker='redis://localhost:6379/0')

@app.task
def add(x, y):
    time.sleep(5)  # Simulate a long-running task
    return x + y

In this code, we define a Celery application and create a task called add that simulates a long-running task by sleeping for 5 seconds.

3. Running the Redis Server

If you haven’t already, run your Redis server to act as the message broker:

redis-server

4. Starting the Celery Worker

To process your tasks, you need to start a Celery worker. Open a new terminal window in the same directory where your tasks.py file is located and run:

celery -A tasks worker --loglevel=info

This starts the Celery worker and allows it to listen for incoming tasks.

5. Calling the Task

In another terminal, you can start a Python shell to call the Celery task:

python

Then run the following commands:

from tasks import add
result = add.delay(4, 4)
print('Task submitted, waiting for result...')
result.ready()  # Check if the task is completed

You can wait for the result using:

print('Result:', result.get())

6. Monitoring Tasks

You can monitor task status and results from the Celery worker terminal, which will display logs for executed tasks, including any errors or completion messages.

7. Conclusion

You have successfully set up Celery for task management in your Python application. Celery allows you to manage and execute background tasks efficiently, enhancing application performance and responsiveness. Continue to explore advanced features of Celery, such as task retries, periodic tasks, and more to make the most of this powerful tool!

About the Author

Dorian Kane

Author

View All Posts