Your API responded in 12 milliseconds. Then a user triggered a PDF export, and suddenly your server is pegged at 100% CPU for eight seconds while everyone else waits. That is not a scaling problem — it is an architecture problem. The fix is a background job queue, and Bull on top of Redis is the most battle-tested solution in the Node.js ecosystem.

Why a Queue, and Why Bull?

Node.js is single-threaded. Long-running tasks — sending a batch of emails, generating reports, calling slow third-party webhooks — block the event loop and degrade every other user's experience. Offloading those tasks to a queue decouples work from the HTTP request cycle entirely.

Bull is a Redis-backed queue library that gives you:

  • Persistence — jobs survive server restarts because they live in Redis, not in memory.
  • Retries with backoff — failed jobs automatically retry with configurable exponential delays.
  • Concurrency control — process exactly as many jobs in parallel as your server can handle.
  • Priority and delay — schedule jobs to run later or promote urgent ones to the front.
  • A built-in UI — via bull-board or arena for monitoring without custom dashboards.

The alternative — setTimeout, in-memory arrays, or fire-and-forget async calls — drops jobs silently on crash. For production SaaS, that is unacceptable.

Setting Up the Stack

You need Redis running. Locally, Docker is the fastest path:

docker run -d -p 6379:6379 redis:7-alpine

Then install Bull and its peer dependency:

npm install bull ioredis

Creating a Queue

A queue is simply a named channel. Create one module per queue type to keep concerns separated.

// queues/emailQueue.js
const Bull = require('bull');

const emailQueue = new Bull('email-notifications', {
  redis: {
    host: process.env.REDIS_HOST || '127.0.0.1',
    port: 6379,
  },
  defaultJobOptions: {
    attempts: 5,
    backoff: {
      type: 'exponential',
      delay: 3000, // 3s, 6s, 12s, 24s, 48s
    },
    removeOnComplete: 100, // keep last 100 completed jobs for auditing
    removeOnFail: 200,
  },
});

module.exports = emailQueue;

Two things worth noting here. First, attempts: 5 with exponential backoff means a transient SMTP failure will not immediately surface as a user-visible error — Bull will quietly retry. Second, removeOnComplete and removeOnFail prevent Redis from filling up with stale job records over time.

Adding Jobs from Your API

Inside your Express (or Fastify) route, add a job instead of doing the work inline:

// routes/orders.js
const emailQueue = require('../queues/emailQueue');

router.post('/orders', async (req, res) => {
  const order = await OrderService.create(req.body);

  // Add to queue — returns immediately
  await emailQueue.add('order-confirmation', {
    to: order.customerEmail,
    orderId: order.id,
  });

  res.status(201).json({ orderId: order.id });
});

The HTTP response goes out in milliseconds. The email sends asynchronously. If Redis is temporarily unavailable, Bull will throw an error you can catch and handle — no silent drops.

Writing the Worker

Workers are separate processes (or at least separate modules) that consume jobs. Running workers as separate Node.js processes means a crash in a worker does not take down your API server.

// workers/emailWorker.js
const emailQueue = require('../queues/emailQueue');
const mailer = require('../services/mailer');

emailQueue.process('order-confirmation', 5, async (job) => {
  const { to, orderId } = job.data;

  await mailer.send({
    to,
    subject: `Your order #${orderId} is confirmed`,
    template: 'order-confirmation',
    context: { orderId },
  });

  return { sentAt: new Date().toISOString() };
});

emailQueue.on('failed', (job, err) => {
  console.error(`Job ${job.id} failed after ${job.attemptsMade} attempts:`, err.message);
  // Send to your error tracker (Sentry, Rollbar, etc.)
});

The 5 in emailQueue.process(name, concurrency, handler) means this worker processes up to five email jobs in parallel. Tune this number based on your SMTP provider's rate limits, not just server capacity.

Handling PDF Generation and Webhooks

The same pattern applies to heavier tasks, with one important adjustment: concurrency should be lower.

PDF generation with tools like Puppeteer is memory-intensive. A concurrency of 2 or 3 is often safer than 10. You can create a dedicated queue:

const pdfQueue = new Bull('pdf-generation', { redis: redisConfig });
pdfQueue.process('export-report', 2, async (job) => { /* ... */ });

For outbound webhooks, the risk is different — third-party endpoints can be slow or return 500s. Exponential backoff is critical here. Add a timeout option per job to prevent a hung HTTP call from blocking a worker slot indefinitely:

webhookQueue.add('delivery', payload, { timeout: 10000 }); // 10s max

Prioritising Jobs

Not all jobs are equal. Bull supports numeric priorities — lower numbers run first:

// Urgent: priority 1
await emailQueue.add('password-reset', data, { priority: 1 });

// Non-urgent: priority 10
await emailQueue.add('weekly-digest', data, { priority: 10 });

This is useful when a single queue handles both transactional and marketing emails — password resets should never wait behind a bulk newsletter.

Monitoring in Production

Add @bull-board/express to mount a read-only dashboard on an authenticated internal route. You will see active, waiting, delayed, completed, and failed jobs in real time. Pair it with Redis memory alerts (set maxmemory-policy to noeviction for queues — you do not want Redis silently discarding jobs as an LRU cache).

Common Pitfalls to Avoid

  • Not separating worker and API processes. A memory leak in a worker should not crash your API.
  • Using the default Redis connection without TLS in production. Always encrypt the connection when Redis is not on localhost.
  • Ignoring the failed event. Unhandled failures are invisible failures. Wire them to your error tracker on day one.
  • Setting removeOnComplete: true (boolean). This removes jobs immediately, leaving you no audit trail. Use a number instead to retain a rolling window.

Why This Matters for Your Project

If you are building a SaaS product that sends notifications, generates documents, syncs data with external APIs, or processes uploads, background jobs are not an optimisation — they are a baseline requirement. Getting this architecture right early means you can scale worker capacity independently of your API servers, replay failed jobs without customer-facing incidents, and confidently add new async features without fear of blocking your core request path. Bull and Redis give you that foundation with minimal operational overhead.