How to Use Celery with RabbitMQ to Queue Tasks on a VPS

Use RabbitMQ as Celery’s broker, run one or more Celery workers as a dedicated systemd service, and submit work with .delay() or .apply_async(). On a Debian or Ubuntu VPS, the reliable production sequence is: install RabbitMQ, create a private user and virtual host, configure an amqp:// broker URL, make tasks retry-safe, then supervise the worker and broker with systemd and normal VPS monitoring.

Antihost offers Linux VPS plans with full root access, unlimited traffic on a 1 Gbps port and a 14-day free trial.

RabbitMQ handles message routing and storage; Celery handles task registration, consumption and execution. Keeping those roles separate makes failures easier to diagnose and lets you scale workers without changing the application that publishes tasks.

#	Preview	Product	Price
1		RabbitMQ in Action: Distributed Messaging for Everyone	$24.98	Buy on Amazon
2		RabbitMQ With C#	$19.95	Buy on Amazon
3		The World of Peter Rabbit (The Original Peter Rabbit, Books 1-23, Presentation Box)	$57.71	Buy on Amazon
4		Hurray for Hattie Rabbit: Story and pictures (An Early I can read book)	$23.87	Buy on Amazon
5		Clojure in Action	$58.99	Buy on Amazon

As an Amazon Associate I earn from qualifying purchases.

What the Celery–RabbitMQ setup does

A web request or another producer sends a task message to RabbitMQ. RabbitMQ places the message in an exchange and queue, then a Celery worker consumes it and calls the Python function registered for that task. The worker acknowledges the message according to Celery’s delivery settings and can write a result to a separate result backend if your application needs one.

Celery identifies RabbitMQ as its default broker and describes it as stable, durable and suitable for production. RabbitMQ also supports queue monitoring and Celery’s remote-control features. The broker is not the worker: installing RabbitMQ alone does not execute Python code, and starting a Celery worker without a reachable broker leaves it unable to receive work.

#1 Best Overall

Sale

RabbitMQ in Action: Distributed Messaging for Everyone

$24.98

Buy on Amazon

Plan the VPS before installing anything

Choose a supported host and account layout

Use a maintained Debian or Ubuntu Linux VPS and apply security updates before deployment. Keep three roles explicit even when they share one machine:

• Application: publishes task messages.
• RabbitMQ: accepts, routes and stores messages.
• Celery worker: consumes messages and executes task code.

Create a non-root Unix account for the application and worker. RabbitMQ has its own service account; do not run your application or worker as root. If the workload grows, the same separation lets you move the broker or add worker VPSs later.

Decide what must survive a restart

Write down whether queued work may be lost during a process, machine or disk failure, how long task results are needed, and how quickly failed jobs must be retried. Celery’s documented default delivery mode is persistent, but persistence is not a single switch: queue and exchange declarations, publisher confirms, broker disk health and result-backend retention all matter.

Install and verify RabbitMQ

1. Update the operating system packages and install RabbitMQ with sudo apt-get update followed by sudo apt-get install rabbitmq-server.
2. Enable and start the service: sudo systemctl enable --now rabbitmq-server.
3. Check the service with sudo systemctl status rabbitmq-server. A running state confirms that systemd started the broker, not that your Celery credentials or queues are correct.
4. Read broker logs if the service fails. Disk alarms, a damaged data directory, an Erlang mismatch or a port already in use must be fixed before you test Celery.

RabbitMQ normally listens for AMQP clients on port 5672. Keep that port on a private interface whenever possible; an internet-facing broker should be treated as an exposed production service, not as a convenience endpoint.

Create a dedicated RabbitMQ user and virtual host

Do not use the default guest account for remote production access. Create a user and a virtual host reserved for this application, then grant permissions only on that virtual host:

sudo rabbitmqctl add_user myuser mypassword
sudo rabbitmqctl add_vhost myvhost
sudo rabbitmqctl set_permissions -p myvhost myuser ".*" ".*" ".*"

Replace the example password with a long, unique secret. The three regular expressions control configuration, write and read permissions. The broad .* patterns are convenient for a single application vhost; a larger installation can narrow them to the exchanges and queues that the application actually uses.

Store the password in an environment variable or a root-readable secrets file rather than in source control. A vhost is part of the connection identity, so the same username can have separate permissions and queues in different vhosts.

Rank #2

RabbitMQ With C#

$19.95

Buy on Amazon

Configure Celery with an AMQP broker URL

Celery’s general broker URL form is transport://userid:password@hostname:port/virtual_host. For a broker on the same VPS, a typical value is:

amqp://myuser:mypassword@localhost:5672/myvhost

In a real deployment, inject this value at runtime:

import os
from celery import Celery

broker_url = os.environ["CELERY_BROKER_URL"]
app = Celery("vps_tasks", broker=broker_url)

If the password contains characters such as @, : or /, URL-encode those characters or use a secrets mechanism that produces a correctly escaped URL. For traffic that crosses an untrusted network, use RabbitMQ TLS and an amqps:// URL, with certificate validation configured to match your security policy. Do not disable certificate checks just to make a connection succeed.

Define a task and publish it asynchronously

Create the Celery application

Put the Celery application in an importable module, such as tasks.py. The module must be available from the worker’s WorkingDirectory and from the process that publishes jobs.

import os
from celery import Celery

app = Celery(
    "vps_tasks",
    broker=os.environ["CELERY_BROKER_URL"],
    backend=os.environ.get("CELERY_RESULT_BACKEND"),
)

app.conf.update(
    task_serializer="json",
    accept_content=["json"],
    result_serializer="json",
    timezone="UTC",
    enable_utc=True,
)

@app.task(bind=True, autoretry_for=(Exception,), retry_backoff=True, max_retries=5)
def process_video(self, video_id):
    # Do work that can safely be repeated after a retry.
    return {"video_id": video_id, "status": "processed"}

JSON serialization keeps task arguments interoperable and avoids sending arbitrary Python objects through the broker. Pass identifiers, strings, numbers and small dictionaries rather than open file handles, database connections or request objects. The task should be idempotent: if a worker crashes after performing the side effect but before acknowledging the message, a redelivery or retry should not corrupt data or charge a customer twice.

Start a worker in the foreground first

From the project directory, activate the same virtual environment used by the application and run:

celery -A tasks worker --loglevel=INFO

The -A tasks argument tells Celery to import the app object from tasks.py. The startup log should show a successful connection to RabbitMQ and the registered task name. Leave this process attached to the terminal while you perform the first test; foreground logs reveal authentication, import and serialization errors immediately.

Rank #3

Sale

The World of Peter Rabbit (The Original Peter Rabbit, Books 1-23, Presentation Box)

• Note: The Hardcover edition (2006) from Warne that you originally viewed is the one you'll receive which is manufactured in china. The Look Inside option is viewing Hardcover edition (2007) from Warne which is printed in Singapore

$57.71

Buy on Amazon

Enqueue work from Python

Call the task from your web application or a one-off producer:

from tasks import process_video

result = process_video.delay("video-123")
print(result.id)

.delay() is shorthand for .apply_async(). Use the longer form when you need execution options:

process_video.apply_async(
    args=["video-123"],
    countdown=30,
    retry=True,
)

The call returns an asynchronous result identifier; it does not mean the task has completed. A result backend is required if you need to query state or return values. Without one, RabbitMQ still transports the task, but your application must record completion in its own database or another system.

Run the worker with systemd

A terminal session, nohup or a shell multiplexer is not a dependable production supervisor. Create a dedicated systemd unit so the worker starts after reboot, restarts after an unexpected exit and runs with a restricted account.

For example, create /etc/systemd/system/celery-worker.service:

[Unit]
Description=Celery worker for the VPS application
Requires=rabbitmq-server.service
After=rabbitmq-server.service network-online.target
Wants=network-online.target

[Service]
Type=simple
User=myapp
Group=myapp
WorkingDirectory=/srv/myapp
EnvironmentFile=/etc/myapp/celery.env
ExecStart=/srv/myapp/.venv/bin/celery -A tasks worker --loglevel=INFO
Restart=on-failure
RestartSec=5

[Install]
WantedBy=multi-user.target

Set ownership and permissions on the environment file so ordinary users cannot read the broker password. The Requires and After lines express the dependency on RabbitMQ; they do not replace health checks or guarantee that the broker is ready to accept connections at the exact instant the worker starts.

1. Reload unit files with sudo systemctl daemon-reload.
2. Start the worker with sudo systemctl start celery-worker.
3. Inspect it using sudo systemctl status celery-worker and sudo journalctl -u celery-worker -f.
4. Enable startup at boot with sudo systemctl enable celery-worker.
5. After changing code or environment settings, run sudo systemctl restart celery-worker and verify that the new task list appears in the log.

Use a separate unit for each materially different queue or workload. A CPU-heavy video task and a latency-sensitive notification task should not necessarily share the same worker pool.

Rank #4

Hurray for Hattie Rabbit: Story and pictures (An Early I can read book)

$23.87

Buy on Amazon

Secure the broker and its network path

• Firewall: permit TCP 5672 only from the application and worker networks. Do not expose the management or AMQP ports to the entire internet.
• Accounts: use a dedicated user and vhost, disable unused accounts, and rotate credentials through the environment or secrets file.
• TLS: use amqps when messages or credentials cross an untrusted network, and validate the server certificate and hostname.
• Updates: patch the operating system, Celery, Kombu, Erlang and RabbitMQ on a planned schedule.
• Backups: back up RabbitMQ definitions and any persistent data according to the recovery point and recovery time your application requires.
• Least privilege: run the worker as a non-root account and restrict its filesystem, database and cloud permissions to the operations its tasks need.

Celery’s security guidance treats an unguarded broker as a critical exposure, especially when it is reachable from a public network. Authentication alone is not a substitute for firewall rules and encrypted transport.

Choose queues and durability deliberately

Persistent messages and classic queues

Persistent delivery asks RabbitMQ to write messages durably, but a durable system also needs durable queue and exchange declarations, healthy broker storage and publisher confirms. A successful publish call without confirmation can still leave an outage window in which the producer does not know whether RabbitMQ accepted the message. Define the failure you are protecting against before increasing durability, because stronger guarantees add disk and operational work.

Quorum queues

Celery supports RabbitMQ quorum queues by declaring x-queue-type: quorum and enabling publisher confirms:

broker_transport_options = {
    "confirm_publish": True,
}

Quorum queues are designed for stronger failure tolerance than a single classic queue, but they change operational behavior. Celery’s current guidance warns that quorum queues disable global QoS and that autoscaling and prefetch-count reduction have limitations when quorum queues are detected. Compare failure tolerance, throughput, disk usage, recovery time and Celery feature compatibility before changing an existing queue type. Queue type is not a cosmetic setting; migrating it normally requires a planned queue migration.

Operate and observe the deployment

Check the three layers separately

• Service state: systemctl status rabbitmq-server and systemctl status celery-worker show whether systemd believes each process is alive.
• Worker logs: journalctl -u celery-worker reveals import failures, rejected messages, retries and task exceptions.
• Broker state: inspect queue depth, publish and delivery rates, consumer counts, disk alarms and memory alarms with RabbitMQ’s administration tools.

A worker can be running while no consumer is attached to the intended queue, and a broker can be healthy while every task is failing in application code. Monitor all three signals rather than treating a green process status as proof that work is completing.

Use graceful shutdowns

Stop a worker with systemd so it can finish or correctly acknowledge work according to your Celery settings. For RabbitMQ maintenance, use a clean service stop such as sudo rabbitmqctl stop or the corresponding systemd action, then verify that the broker has shut down before performing storage or package operations. Automatic restart is useful for transient failures but is not a replacement for monitoring, alerting and a recovery plan.

Scale workers without losing control

Start with one worker and measure queue depth, task runtime, CPU, memory and disk I/O. Add worker processes or additional VPSs when the workload requires more concurrency, but keep the broker on a machine with predictable disk latency. Separate long-running and short-running jobs into named queues, then bind dedicated workers to those queues. This prevents a backlog of large jobs from starving quick tasks.

Best Value

Clojure in Action

$58.99

Buy on Amazon

Set concurrency and prefetch conservatively. High prefetch can improve throughput for tiny tasks but can also let one worker reserve many messages, increasing latency and making an abrupt worker loss more disruptive. Quorum queues require special attention because global QoS is disabled and some prefetch-reduction behavior is limited.

Make retries bounded and observable. Exponential backoff, a maximum retry count and an error path prevent a broken downstream service from creating an endless storm. Record a task identifier and business identifier in logs so an operator can correlate the producer request, broker message and worker exception.

Troubleshoot common failures

“Connection refused” or timeouts

Confirm that RabbitMQ is active, that the hostname and port in CELERY_BROKER_URL are correct, and that the VPS firewall permits the worker’s source address. If RabbitMQ is on another host, test the private network route and TLS mode. A local localhost URL is wrong when the worker has been moved to a different VPS.

“Authentication failure” or “access refused”

Check the username, URL-encoded password and vhost spelling. Verify the account’s permissions with RabbitMQ administration commands. The vhost is part of the URL; authenticating successfully to the default vhost does not grant access to myvhost.

The worker starts but no tasks arrive

Compare the task name registered in the worker startup log with the name published by the application. Confirm that both processes use the same broker URL and queue configuration. Check queue depth and consumer count. An application may be publishing successfully to one vhost or queue while the worker consumes another.

Tasks run twice

At-least-once delivery means a crash around acknowledgement can cause redelivery. Make the task idempotent, use a durable business-side deduplication key where needed, and avoid placing non-repeatable side effects before the operation that records completion. Do not “fix” duplicates by disabling acknowledgements without understanding the loss trade-off.

Messages disappear after a restart

Verify that the queue and exchange are durable, messages are published with persistent delivery, publisher confirms are enabled where required, and RabbitMQ’s data directory is healthy. A result backend may also expire results even though the original task message was durable; message retention and result retention are separate policies.

Memory or disk alarms pause publishing

Inspect RabbitMQ alarms and free disk space before restarting services repeatedly. Reduce backlog, correct the underlying resource limit and review task payload size. Large binary files should normally be stored outside the broker, with the task carrying an object identifier or path rather than the file contents.

RabbitMQ compared with other Celery brokers

Celery’s broker overview marks RabbitMQ and Redis as stable, and also marks Amazon SQS as stable. They are not interchangeable operationally:

For a single VPS where you want predictable Celery features and direct operational control, RabbitMQ is a defensible default. Choose an alternative because its delivery model and operating burden fit your workload, not because changing the URL is considered a complete migration plan.

Quick Recap

SaleBestseller No. 1

RabbitMQ in Action: Distributed Messaging for Everyone

$24.98

Bestseller No. 2

RabbitMQ With C#

$19.95

SaleBestseller No. 3

The World of Peter Rabbit (The Original Peter Rabbit, Books 1-23, Presentation Box)

$57.71

Bestseller No. 4

Hurray for Hattie Rabbit: Story and pictures (An Early I can read book)

$23.87

Bestseller No. 5

Clojure in Action

$58.99

Production checklist

• Debian or Ubuntu is patched and the application runs under a non-root account.
• RabbitMQ is enabled, healthy and reachable only from approved networks.
• A dedicated RabbitMQ user and vhost replace remote use of guest.
• The broker URL is injected through protected configuration, not committed to source control.
• Tasks use serializable arguments, bounded retries and idempotent side effects.
• The worker is supervised by systemd with User, Group, WorkingDirectory and boot-time enablement.
• Queue depth, consumer count, task failures, memory alarms and disk alarms are monitored.
• Persistence, publisher confirms, result retention and recovery procedures are documented.
• Quorum queues are adopted only after checking their QoS and autoscaling implications.
• Credentials, certificates, RabbitMQ definitions and persistent data have a rotation and backup plan.

Product prices and availability are accurate as of the date/time indicated and are subject to change. Any price and availability information displayed on Amazon at the time of purchase will apply.