Designing a Rate Limiter in Node.js Without a Cache Layer

Every production API eventually meets an abusive client — a misconfigured mobile app hammering an endpoint, a scraper, or a legitimate user whose loop went rogue. Rate limiting is the first line of defense, and the standard advice is always the same: "just use Redis." That advice is correct at scale. But for a small SaaS team in Accra or Lagos running a single Node.js process on a $10 VPS or a modest cloud instance, spinning up and paying for a managed Redis cluster to protect a few endpoints is genuine over-engineering. In-process rate limiting deserves a proper look — including an honest accounting of where it eventually fails.

Two Algorithms Worth Understanding

Before writing a single line of code, you need to pick an algorithm. The two most practical for in-process use are the token bucket and the sliding window counter.

Token Bucket

Imagine a bucket that holds a fixed number of tokens. Each incoming request consumes one token. Tokens are refilled at a constant rate. If the bucket is empty, the request is rejected. This model is naturally forgiving of short bursts — a client can fire several requests quickly as long as tokens are available — while still enforcing a long-term average rate.

Sliding Window Counter

Rather than tokens, you track the timestamps of recent requests within a rolling time window. A request is allowed if the count of requests in the last N milliseconds is below the limit. It is more memory-intensive per client but produces a smoother rate curve and prevents the "reset cliff" problem where a fixed-window counter resets and briefly allows a doubled burst.

For most SaaS APIs, the sliding window is the fairer algorithm for end users. The token bucket is easier to reason about and slightly cheaper in CPU terms.

Building a Token Bucket in Node.js

Here is a minimal, production-usable token bucket implemented as a plain JavaScript class — no dependencies, no Redis, no magic.

class TokenBucket {
  constructor(capacity, refillRatePerSecond) {
    this.capacity = capacity;
    this.refillRatePerSecond = refillRatePerSecond;
    this.buckets = new Map(); // keyed by client identifier
  }

  _getBucket(key) {
    const now = Date.now();
    if (!this.buckets.has(key)) {
      this.buckets.set(key, { tokens: this.capacity, lastRefill: now });
    }
    const bucket = this.buckets.get(key);
    const elapsed = (now - bucket.lastRefill) / 1000;
    bucket.tokens = Math.min(
      this.capacity,
      bucket.tokens + elapsed * this.refillRatePerSecond
    );
    bucket.lastRefill = now;
    return bucket;
  }

  consume(key) {
    const bucket = this._getBucket(key);
    if (bucket.tokens >= 1) {
      bucket.tokens -= 1;
      return true; // request allowed
    }
    return false; // request denied
  }
}

// Express middleware usage
const limiter = new TokenBucket(20, 5); // 20-token capacity, 5 tokens/sec refill

app.use((req, res, next) => {
  const clientKey = req.ip; // or req.user?.id for authenticated routes
  if (!limiter.consume(clientKey)) {
    return res.status(429).json({ error: "Too many requests. Slow down." });
  }
  next();
});

This runs entirely in memory. No network round-trips, no serialization overhead, sub-millisecond enforcement. On a modest server, this implementation will comfortably handle thousands of requests per second.

The Memory Leak You Must Address

The Map in the example above grows indefinitely if you never clean it up. Each unique IP or user ID that ever hits your API adds an entry that never expires. On a public-facing endpoint, this will eventually exhaust heap memory.

The fix is a periodic cleanup pass that evicts stale buckets:

setInterval(() => {
  const now = Date.now();
  const staleThreshold = 60_000; // 60 seconds of inactivity
  for (const [key, bucket] of limiter.buckets) {
    if (now - bucket.lastRefill > staleThreshold) {
      limiter.buckets.delete(key);
    }
  }
}, 30_000);

Run this every 30 seconds and your memory footprint stays bounded. This is not optional — it is table stakes for any in-process rate limiter in production.

Where This Approach Breaks Down

Being honest about failure modes is what separates engineering from tutorials. In-process rate limiting has three hard limitations:

1. It does not survive horizontal scaling. The moment you run two Node.js instances behind a load balancer — whether for redundancy or traffic — each process maintains its own independent state. A client hitting both instances effectively gets double the rate limit. No amount of clever in-process code fixes this; you need a shared state store (Redis, Memcached, a database) the instant you scale beyond one process.

2. It does not survive process restarts. A crash, a deployment, or a server reboot wipes all in-memory rate limit state. Clients that were throttled seconds ago get a fresh slate. For most SaaS use cases this is an acceptable tradeoff, but it is worth acknowledging.

3. It is vulnerable to coordinated abuse across IPs. In-process limiting keyed by IP works well against clumsy abuse. Against a distributed botnet rotating through hundreds of IP addresses, it offers no protection. That requires application-level anomaly detection, not a rate limiter.

When In-Process Rate Limiting Is the Right Call

Despite the limitations, there are concrete scenarios where this approach is exactly correct:

  • Single-server deployments serving a focused user base — common for early-stage SaaS products, internal tools, or MVPs
  • Protecting expensive endpoints (report generation, bulk exports) from accidental overuse rather than malicious attack
  • Development and staging environments where adding Redis just to test rate limiting adds friction
  • Edge functions or serverless runtimes where a full Redis connection is impractical or cost-prohibitive per invocation

The pattern also composes well. Many teams run an in-process limiter as a first gate — cheap and fast — alongside a Redis-backed limiter that only activates at the cluster level once they scale. You pay for Redis only when it actually buys you something.

Practical Additions Before You Ship

A bare token bucket is functional, but production use calls for a few enhancements:

  • Return standard headers: X-RateLimit-Limit, X-RateLimit-Remaining, and Retry-After tell API consumers exactly what is happening and make debugging dramatically easier.
  • Differentiate limits by route: A login endpoint warrants far stricter limits (e.g., 5 requests/minute) than a public product listing endpoint.
  • Log rejections: Every 429 response is a signal. Aggregate them. A spike in rate limit rejections is often the first indicator of a bug in a client SDK, a credential stuffing attempt, or a viral traffic event.
  • Test under load: Use a tool like autocannon or k6 to verify your limits hold under concurrency before you push to production.

Why This Matters for Your Project

Rate limiting is not glamorous infrastructure, but it is one of the cheapest insurance policies available to a SaaS backend. Starting with an in-process implementation means you ship protection on day one, with zero additional cost or operational overhead. The discipline of understanding exactly where that implementation breaks — and having a clear migration path to a shared store when you need it — is precisely the kind of technical clarity that separates teams that scale gracefully from teams that scramble. Build the simple thing first. Know when to replace it.