When MoMo or Paystack returns a 503 at 2 a.m., your app shouldn't just swallow the error. Here's how to build a production-grade retry queue with exponential backoff, dead-letter logging, and alerting

How to Build a Retry Queue for Failed API Calls in Node.js

A payment initiation to MTN MoMo returns a 503. Paystack's webhook delivery times out. Your Flutterwave disbursement call drops midway through a batch. These are not edge cases — they are Tuesday morning in production.

Most tutorials walk you through the happy path: make a call, get a 200, move on. Real backend engineering starts at the failure boundary. This article walks through building a production-grade retry queue in Node.js — one that handles silent failures, applies exponential backoff, logs dead letters, and fires alerting hooks when things go persistently wrong.

Why a Simple try/catch Is Not Enough

A naive implementation catches the error and either swallows it or immediately throws it back to the user. Neither is acceptable when money is on the line.

The real problems with third-party API failures are:

Silent failures — the API returns 200 but the transaction is in a pending state that never resolves.
Transient errors — a 429 (rate limit) or 503 (service unavailable) that would succeed if retried 10 seconds later.
No audit trail — when a failure slips through, there is nothing to investigate.
No alerting — your team finds out when a customer calls, not when the error happens.

A retry queue addresses all four.

The Architecture at a Glance

The system has four moving parts:

Job producer — pushes a failed API call into a queue with metadata.
Worker — pulls jobs, attempts the call, applies backoff between retries.
Dead-letter store — permanently logs jobs that exhaust all retries.
Alerting hook — fires a notification (Slack, PagerDuty, SMS) when a job dies.

For the queue itself, you can use BullMQ (backed by Redis) for production. For teams who cannot add Redis to their stack yet, an in-process queue with p-queue works for low-volume scenarios — but Redis-backed is strongly recommended for anything handling financial transactions.

Setting Up BullMQ

npm install bullmq ioredis axios

// queues/apiRetryQueue.js
import { Queue, Worker, QueueEvents } from "bullmq";
import IORedis from "ioredis";

const connection = new IORedis(process.env.REDIS_URL, {
  maxRetriesPerRequest: null,
});

// Create the queue
export const apiRetryQueue = new Queue("api-retry", { connection });

// Producer: enqueue a failed job
export async function enqueueApiCall(jobName, payload) {
  await apiRetryQueue.add(jobName, payload, {
    attempts: 5,
    backoff: {
      type: "exponential",
      delay: 3000, // starts at 3s, then 6s, 12s, 24s, 48s
    },
    removeOnComplete: true,
    removeOnFail: false, // keep failed jobs for inspection
  });
}

BullMQ handles the exponential backoff math for you when type: "exponential" is set. Each retry doubles the previous delay, giving the upstream API time to recover without hammering it.

Writing the Worker

The worker is where your actual API call lives. It receives the job payload and attempts the request.

// workers/apiRetryWorker.js
import { Worker } from "bullmq";
import axios from "axios";
import { connection } from "../queues/apiRetryQueue.js";
import { logDeadLetter } from "../services/deadLetterLogger.js";
import { fireAlert } from "../services/alerting.js";

const worker = new Worker(
  "api-retry",
  async (job) => {
    const { endpoint, method, body, headers } = job.data;

    try {
      const response = await axios({ url: endpoint, method, data: body, headers, timeout: 10000 });

      // Validate the response — don't trust a 200 blindly
      if (response.data?.status === "pending" && !response.data?.transactionId) {
        throw new Error("Silent failure: transaction pending with no ID returned");
      }

      return response.data;
    } catch (error) {
      const isLastAttempt = job.attemptsMade >= job.opts.attempts - 1;

      if (isLastAttempt) {
        await logDeadLetter(job, error);
        await fireAlert({
          message: `API job "${job.name}" permanently failed after ${job.opts.attempts} attempts.`,
          jobData: job.data,
          error: error.message,
        });
      }

      // Re-throw so BullMQ knows the attempt failed and schedules a retry
      throw error;
    }
  },
  { connection, concurrency: 5 }
);

Notice the silent failure check. MoMo and similar APIs sometimes return HTTP 200 with a transaction stuck in pending. Treating that as success is how you lose money. Always validate the shape and semantics of the response, not just the status code.

Dead-Letter Logging

A dead-letter store is a permanent record of every job that could not be processed after all retries. It is your audit trail, your debugging surface, and — in regulated fintech environments — potentially a compliance requirement.

// services/deadLetterLogger.js
import fs from "fs/promises";
import path from "path";

export async function logDeadLetter(job, error) {
  const entry = {
    timestamp: new Date().toISOString(),
    jobId: job.id,
    jobName: job.name,
    attemptsMade: job.attemptsMade,
    data: job.data,
    errorMessage: error.message,
    stack: error.stack,
  };

  // In production, write to a DB table or a log aggregator (Datadog, Logtail, etc.)
  const logPath = path.resolve("logs/dead-letters.ndjson");
  await fs.appendFile(logPath, JSON.stringify(entry) + "\n");
}

For production, replace the file append with an insert into a dead_letters database table, or ship the entry to your log aggregator. The pattern is the same — the destination changes.

Alerting Hooks

Logging is passive. Alerting is active. Your on-call engineer needs to know within minutes, not hours.

// services/alerting.js
import axios from "axios";

export async function fireAlert({ message, jobData, error }) {
  const slackPayload = {
    text: `*[DEAD LETTER ALERT]* ${message}`,
    attachments: [
      {
        color: "danger",
        fields: [
          { title: "Endpoint", value: jobData.endpoint, short: true },
          { title: "Error", value: error, short: false },
        ],
      },
    ],
  };

  await axios.post(process.env.SLACK_WEBHOOK_URL, slackPayload);
  // Add PagerDuty, SMS via Twilio, or email here as needed
}

Composing multiple alerting channels is straightforward — call each service in sequence or with Promise.allSettled so one failing channel does not block the others.

Plugging It Into Your Integration Flow

When a Paystack charge call fails, instead of throwing directly to the user:

try {
  await initiatePaystackCharge(payload);
} catch (err) {
  await enqueueApiCall("paystack-charge", {
    endpoint: "https://api.paystack.co/charge",
    method: "POST",
    body: payload,
    headers: { Authorization: `Bearer ${process.env.PAYSTACK_SECRET}` },
  });
  // Return a 202 Accepted to the client — the job will be processed asynchronously
  return res.status(202).json({ message: "Payment is being processed. You will be notified." });
}

This shifts the user experience from a hard failure to a graceful async acknowledgment — a significant UX and reliability improvement.

Key Configuration Decisions

Parameter	Recommended Value	Reasoning
Max attempts	5	Covers transient outages without running indefinitely
Initial backoff delay	3,000 ms	Enough breathing room for rate-limit recovery
Concurrency	3–10	Balance throughput against upstream rate limits
Job TTL	24–48 hours	Prevents Redis bloat from stale jobs

Why This Matters for Your Project

Every SaaS or fintech product operating in Ghana or across Africa will interact with at least one payment API that has variable uptime. A retry queue is not an optimisation — it is a reliability primitive. Without it, silent failures erode customer trust, create reconciliation nightmares, and introduce revenue leakage that is often invisible until it compounds. Building this pattern early, before scale forces the conversation, is one of the clearest signs of a mature engineering team.