
## Introduction

[Celery](https://github.com/celery/celery) is an open-source distributed task queue that focuses on real-time processing and task scheduling. It coordinates tasks through a backing message broker, typically [Redis](https://redis.io/) or [RabbitMQ](https://www.rabbitmq.com/), making it straightforward to scale as demand increases. While Celery is written in Python, its messaging protocols are not language specific, allowing for clients in any language.

In this guide, we will go over how to deploy a Celery worker on the [Koyeb](/) serverless platform using git-driven deployment. We will create a repository with Celery and its dependencies installed, deploy RabbitMQ to act as our backing message broker, and then connect our GitHub account to Koyeb to automatically deploy the Celery worker to the platform.

At the end of this guide, you should have a Celery worker deployed to Koyeb that you can automatically update by pushing changes to your GitHub account.

## Requirements

To follow along with this guide, you will need the following:

- Python 3 installed on your local machine
- A [Koyeb account](https://app.koyeb.com/) to deploy and run your Celery worker and RabbitMQ message broker
- A [GitHub account](https://github.com/) to store your application code and trigger deployments via repository changes

## Steps

This guide will cover how to deploy a Celery worker to Koyeb through the following steps:

1. [Create a Python Virtual Environment](#create-a-python-virtual-environment)
2. [Install Celery](#install-celery)
3. [Create a Basic Celery Application](#create-a-basic-celery-application)
4. [Push the Celery Project to GitHub](#push-the-celery-project-to-git-hub)
5. [Deploy RabbitMQ on Koyeb](#deploy-rabbit-mq-on-koyeb)
6. [Deploy the Celery Worker on Koyeb](#deploy-the-celery-worker-on-koyeb)
7. [Testing the Celery Worker](#testing-the-celery-worker)

## Create a Python Virtual Environment

To get started, we will create a Python virtual environment on our local machine to isolate our project's dependencies from the rest of the system packages. Python provides this functionality natively through the standard [`venv` module](https://docs.python.org/3/library/venv.html).

Start by creating a new directory to store our project files.

```shell
mkdir ~/celery-koyeb
```

Next, create the virtual environment within the project directory by typing:

```shell
python3 -m venv ~/celery-koyeb/venv
```

This will create a directory called `venv` within the `~/celery-koyeb` containing initialization scripts, copies of essential tools like `pip`, and links to the local Python interpreters.

To tell Python to use the virtual environment, activate it by sourcing the initialization script:

```shell
source ~/celery-koyeb/venv/bin/activate
```

Your shell prompt will change to reflect that the virtual environment is active for this session. Python will now use this directory structure in place of the system's standard Python file locations, allowing us to isolate dependencies.

## Install Celery

Next, we need to install Celery within the virtual environment using `pip`, the Python package manager:

```shell
pip install celery
```

Python will download Celery and its dependencies and install them within the virtual environment filesystem.

Along with the Python modules, the installation also includes a small wrapper script called `celery` that lets you interact with it from the command line. Use the command now to verify that the installation was successful:

```shell
celery --version
```

The command should show the version as expected. The exact version returned does not matter, so long as the command returns successfully:

```
5.2.7 (dawn-chorus)
```

With Celery installed, record the project's dependencies by typing:

```shell
pip freeze > ~/celery-koyeb/requirements.txt
```

## Create a Basic Celery Application

Next, we'll create a basic Celery application in our project directory.

Create a file called `tasks.py` within the `~/celery-koyeb` directory with the following content inside:

```python
import os
from celery import Celery


# Fail if the `BROKER_HOST` environment variable is not defined
BROKER_HOST = os.environ.get('BROKER_HOST', None)
if BROKER_HOST is None:
    raise Exception("BROKER_HOST environmental variable is not defined")
broker_url = 'pyamqp://guest@' + BROKER_HOST + '//'

app = Celery('tasks', broker=broker_url, backend='rpc://')

@app.task
def add(x, y):
    return x + y
```

Let's talk a bit about what is happening in the code above.

After importing the `os` module and the `Celery` class, the code above sets the location of the backing RabbitMQ message broker based on the value of the `BROKER_HOST` environment variable. If this variable is not set, we will error out to avoid accidentally deploying misconfigured instances.

Next, we create an instance of the `Celery` class called `app`. We pass `tasks` as the name of the main module and include the broker and backend configuration to allow Celery to connect to the RabbitMQ instance indicated by the environment variable mentioned earlier.

Finally, we create a function with the `@app.task` decorator to add it to Celery's task registry. Our function adds two numbers together and returns the result.

Save and close the file when you are finished.

## Push the Celery Project to GitHub

Next, we will commit our project to `git` and push our changes to our GitHub account.

In your project directory, initialize a `git` repository to get started:

```shell
cd ~/celery-koyeb
git init
```

Add the `tasks.py` application and the `requirements.txt` file to the staging area. You can also create a minimal `.gitignore` file to make sure your virtual environment does not accidentally get committed. Commit the changes when you are done:

```shell
echo 'venv' >> .gitignore
git add tasks.py requirements.txt .gitignore
git commit -m 'Initial commit'
```

Create a [new repository](https://github.com/new) within your GitHub account. This can be either public or private. Add the GitHub repository address as the `origin` of your project and push your changes to the repository:

```shell
git remote add origin git@github.com:<YOUR_GITHUB_USERNAME>/<YOUR_GITHUB_REPOSITORY>.git
git branch -M main
git push -u origin main
```

If you refresh the GitHub repository page in your browser, your project files will appear.

## Deploy RabbitMQ on Koyeb

Before we deploy our Celery worker, we need to deploy a RabbitMQ instance for Celery to use as a message broker. We will use the [official RabbitMQ image on DockerHub](https://hub.docker.com/_/rabbitmq) for this purpose.

Go to the **Overview** tab of the [Koyeb control panel](https://app.koyeb.com/) and click **Create Web Service** to begin:

1. Select **Docker** as the deployment method.
2. In the **Docker image** field, enter `docker.io/rabbitmq`.
3. In the **Exposed ports** section, change the port number to 5672 and deselect the **Public** option. This will allow other services within this application (like our Celery application) to access the message broker using its service name without exposing the service to the internet.
4. Click the **Deploy** button.

Your new application will be created and the RabbitMQ service will begin to deploy within it. In a few minutes, the RabbitMQ container will finish deploying to Koyeb and the service will be marked as healthy.

Before moving on, copy the **Private domain** of your RabbitMQ deployment. We will need this later as we configure the Celery worker service.

## Deploy the Celery Worker on Koyeb

Now that the RabbitMQ service is deployed, we can deploy the Celery application from our GitHub repository.

In the **Apps** tab of the [Koyeb control panel](https://app.koyeb.com/), click the app that contains your RabbitMQ service. Click the **Create Service** button to deploy a new service to the app alongside the RabbitMQ container:

1. Choose **GitHub** as the deployment method.
2. Select the repository for your Celery application from the list.
3. In the **Service type** section, select **Worker**.
4. In the **Builder** section, click the **Override** toggle associated with the **Run command** and enter `celery --app=tasks.app worker` in the field.
5. In the **Environment variables** section, create a new variable called `BROKER_HOST` to tell the Celery worker where to find the RabbitMQ instance. Set the value to the **Private domain** you copied from the RabbitMQ service. It will follow this format: `<YOUR_SERVICE_NAME>.<YOUR_APP_NAME>.koyeb`.
6. Click the **Deploy** button.

The Celery application will be pulled from your GitHub repository and deployed. In a few minutes, the Celery worker will begin running within the service and be available for work.

If you want to learn about how Koyeb automatically builds your Python applications from git, make sure to read the [how we build from git](/docs/apps/build-from-git) documentation.

## Testing the Celery Worker

Unlike a web application, Celery does not have a web interface to easily show you that the deployment succeeded. We can, however, use Koyeb's built-in Console to test that the worker is running, able to connect with RabbitMQ, and can process tasks.

In the [Koyeb control panel](https://app.koyeb.com/), click the Celery service and then select the **Console** tab.

In the console window, start a `bash` session by typing:

```shell
/bin/bash
```

Next, source the environment configuration file so that Python can find the modules associated with our application:

```shell
source .profile.d/python.sh
```

Start a Python session so that we can interact with the Celery application:

```shell
python
```

First, import our `tasks` module:

```python
import tasks
```

Next, we can run the `add` function using [Celery's `delay` method](https://docs.celeryq.dev/en/stable/reference/celery.app.task.html#celery.app.task.Task.delay):

```python
results = tasks.add.delay(93, 80)
```

We can then use the `results` object to check on the task's progress. Call the [`ready` method](https://docs.celeryq.dev/en/stable/reference/celery.result.html#celery.result.AsyncResult.ready) to see if it has been processed yet:

```python
results.ready()
True
```

You can check the answer returned by the function using the [`get` method](https://docs.celeryq.dev/en/stable/reference/celery.result.html#celery.result.AsyncResult.get). It's usually a good idea to set a timeout so that it won't hang waiting if the result is not available yet:

```python
results.get(timeout=1)
173
```

While this test uses trivial calculations, the results validate that the system is working as intended and able to communicate with RabbitMQ using Koyeb's [service mesh](/docs/reference/service-mesh-and-discovery).

When you are finished, exit the Python session by typing:

```python
exit()
```

You can then close the bash session by typing:

```shell
exit
```

## Conclusion

In this guide, we covered how to deploy a Celery worker on Koyeb using git-driven deployment. Because Celery requires a message broker for coordination, we also discussed how to deploy a RabbitMQ container on Koyeb using Docker-based deployment.

By deploying both of these services within a single application, we were able to configure Celery to connect with RabbitMQ over Koyeb's service mesh with minimal configuration. Our Celery deployment is backed by a GitHub repository, so it will automatically redeploy whenever new changes are pushed to the branch it is tracking. This makes it easy to change your running Celery worker configuration by modifying your project's code.

Questions or suggestions to improve this guide? Join us on the [community platform](https://community.koyeb.com/) to chat!


Dynamically scale your Celery workers with ease on Koyeb. Run on high-performance infrastructure. No credit card required.

Easy-to-scale Celery workers

This guide explains how to deploy Python Celery Workers using git-driven deployment on Koyeb Serverless Platform.

Asynchronous

Deploy a Python Celery Worker