Idempotency: A Beginner's Guide to Safe Retries in APIs

Idempotency is the property that makes running the same operation multiple times produce the same result as running it once. It is load-bearing for any API that handles money, side effects, or distributed work, because the underlying transport (HTTP, mobile radios, message brokers) retries on failure regardless of intent. A non-idempotent endpoint turns every retry into a duplicate write: double charges, duplicate orders, jobs fulfilled twice.

This guide is for developers building APIs, payment flows, or message consumers. It covers HTTP method semantics, the idempotency key pattern used by Stripe and others, database-level upserts, and the pitfalls (cache mismatches, key collisions, non-deterministic side effects) that turn safe-looking idempotency into a quiet source of bugs.

What Idempotency Means

An operation is idempotent if running it multiple times produces the same result as running it once. The math definition is f(f(x)) = f(x). In practice: if you call the same endpoint twice with the same input, the system ends up in the same state as if you called it once.

Note the subtlety. Idempotency is not about blocking duplicate requests. It is about making sure duplicates do not cause duplicate effects. The network will retry whether you like it or not. Your job is to tolerate it.

Three Related Terms

• Safe: no side effects at all (a GET request)
• Idempotent: side effects happen, but repeats add nothing new (PUT, DELETE)
• Pure: deterministic, no side effects, no external state (math functions)

Every safe operation is idempotent. Not every idempotent operation is safe.

Why It Matters: The Double-Charge Problem

Consider a checkout flow:

The first charge succeeded on the server, but the response never reached the client. The client retried, and now the customer has been charged twice. The server had no way to know the second request was a retry of the first.

The fix is an idempotency key: a unique ID the client generates once and reuses across retries of the same logical operation.

HTTP Methods and Idempotency

RFC 9110 defines the contract for standard HTTP methods:

Method	Safe	Idempotent	Typical Use
GET	Yes	Yes	Read data
PUT	No	Yes	Full replacement
DELETE	No	Yes	Remove resource
POST	No	No	Create new resource
PATCH	No	Depends	Partial update

A PUT /users/123 with the same body, called three times, leaves the user record in the same state. A DELETE /orders/456 called twice still results in the order being gone. But POST /orders creates a new order each time, so it is not idempotent by default.

This matters because browsers, proxies, and HTTP clients can retry GET, PUT, and DELETE requests automatically, assuming they are safe. If you use POST where PUT would fit, you lose that guarantee.

The Idempotency Key Pattern

This is the standard way to make POST operations idempotent, popularized by Stripe.

How It Works

1. The client generates a UUID before sending the request.
2. The client sends it in an Idempotency-Key header.
3. The server checks a fast store (Redis, DynamoDB) for that key.
4. If the key is new, the server processes the request and stores the full response with a TTL.
5. If the key already exists, the server returns the stored response without re-running the business logic.

A Minimal Express Implementation

import express, { Request, Response, NextFunction } from "express";
import { createClient } from "redis";
import { randomUUID } from "crypto";

const redis = createClient();
await redis.connect();

interface StoredResponse {
  status: number;
  body: unknown;
}

async function idempotency(req: Request, res: Response, next: NextFunction) {
  const key = req.header("Idempotency-Key");
  if (!key) return next();

  // Scope by user and endpoint to avoid cross-tenant collisions
  const storeKey = `idem:${req.user?.id}:${req.path}:${key}`;

  const cached = await redis.get(storeKey);
  if (cached) {
    const stored: StoredResponse = JSON.parse(cached);
    return res.status(stored.status).json(stored.body);
  }

  // Lock the key for 30 seconds to handle concurrent retries
  const locked = await redis.set(storeKey + ":lock", "1", {
    NX: true,
    EX: 30,
  });
  if (!locked) {
    return res.status(409).json({ error: "Request in progress" });
  }

  // Capture the response so we can store it
  const originalJson = res.json.bind(res);
  res.json = (body: unknown) => {
    const toStore: StoredResponse = { status: res.statusCode, body };
    // 24 hour TTL, matching Stripe's default
    redis.set(storeKey, JSON.stringify(toStore), { EX: 86400 });
    return originalJson(body);
  };

  next();
}

This middleware handles the essentials: scoping by user, locking for concurrency, storing the full response, and replaying on retry. Production systems usually add more: distinguishing in-progress from completed, validating that request bodies match the stored key, and richer error handling.

Client Side

import { randomUUID } from "crypto";

async function chargeCustomer(amount: number) {
  const idempotencyKey = randomUUID();

  // Reuse the same key across all retries of this logical operation
  for (let attempt = 0; attempt < 3; attempt++) {
    try {
      const res = await fetch("/api/charge", {
        method: "POST",
        headers: {
          "Content-Type": "application/json",
          "Idempotency-Key": idempotencyKey,
        },
        body: JSON.stringify({ amount }),
      });
      if (res.ok) return res.json();
    } catch (err) {
      // Network error, retry with the same key
      await new Promise((r) => setTimeout(r, 1000 * (attempt + 1)));
    }
  }
  throw new Error("Charge failed after retries");
}

The key point: the client generates the key once, before the first attempt, and reuses it on every retry. A new UUID on each attempt would defeat the entire pattern.

Database-Level Idempotency

Sometimes the database can do the work for you. Unique constraints and conditional writes give you idempotency with almost no application code.

PostgreSQL Upsert

INSERT INTO orders (id, user_id, amount, created_at)
VALUES ($1, $2, $3, NOW())
ON CONFLICT (id) DO NOTHING
RETURNING *;

If the caller provides the order ID, running this twice leaves the same row in the table. The second call returns nothing because of DO NOTHING, which your handler can treat as a successful no-op.

DynamoDB Conditional Write

import { DynamoDBClient, PutItemCommand } from "@aws-sdk/client-dynamodb";

const client = new DynamoDBClient({});

async function createOrder(orderId: string, data: Record<string, string>) {
  try {
    await client.send(
      new PutItemCommand({
        TableName: "Orders",
        Item: {
          id: { S: orderId },
          data: { S: JSON.stringify(data) },
        },
        ConditionExpression: "attribute_not_exists(id)",
      }),
    );
    return { created: true };
  } catch (err: any) {
    if (err.name === "ConditionalCheckFailedException") {
      // Already exists, treat as success
      return { created: false };
    }
    throw err;
  }
}

The attribute_not_exists condition makes the write succeed only the first time. Retries land in the catch block and become a no-op.

Idempotency in Message Queues

Most queues guarantee at-least-once delivery, not exactly-once. SQS, Kafka, RabbitMQ, and Pub/Sub can all deliver the same message more than once. Consumers crash before acknowledging, visibility timeouts expire, producers retry. Your consumer must tolerate replays.

A simple dedup table keyed by message ID, with a TTL slightly longer than your retention window, is often enough. For stronger guarantees, combine the side effect and the “processed” marker in one database transaction (the outbox pattern).

Exactly-Once Is a Myth

Exactly-once delivery is impossible in distributed systems. This is a theoretical result (the Two Generals Problem, if you want to read more). What you can achieve is exactly-once processing, by combining at-least-once delivery with idempotent handlers:

at-least-once delivery + idempotent processing = effectively exactly-once

Kafka’s “exactly-once semantics” works within the Kafka ecosystem, but the moment you send an email or call an external API, you are back to needing idempotent handlers.

Common Pitfalls

These are the mistakes that break idempotency in practice.

1. Using Wall-Clock Time Inside the Handler

If your handler computes created_at = NOW() on each call, the stored rows will differ between the first call and a retry. The operation is no longer idempotent in the strict sense.

Fix: capture timestamps once and store them with the idempotency key, or pass them as parameters.

2. Side Effects Outside the Idempotency Boundary

A common pattern: commit to the database, then send an email. If the email send fails and the client retries, the database write succeeds twice (if not guarded) or the email fires twice (if it is).

Fix: write the email intent into the database inside the same transaction, and have a separate worker deliver it idempotently.

3. Timestamps as Idempotency Keys

Keys like user-123-1696000000 collide under load and break under clock drift. Use UUID v4 or v7. Never rely on wall-clock time alone.

4. Forgetting to Store the Response Body

Marking a key as “processed” without storing the response leaves you unable to replay. The retry either errors or re-runs the logic.

Fix: store the full response (status, headers, body) atomically with the processed marker.

5. Ignoring Concurrency

Two simultaneous requests with the same key both miss the cache, both run the handler, both store a result. One wins, but both side effects happened.

Fix: use a lock or a database unique constraint on the key during the “mark as processing” step.

6. Leaking Keys Across Tenants

A global key store without scoping lets one customer’s key match another’s operation.

Fix: scope keys as tenant_id:user_id:endpoint:key.

7. TTL Too Short

If keys expire before the client gives up, a late retry causes a second execution.

Fix: pick a TTL longer than any reasonable retry window. 24 hours is a sensible default.

When to Use What

• Read-only endpoint: use GET. Nothing more needed.
• Full replacement: use PUT. Idempotent by contract.
• Resource removal: use DELETE. Idempotent by contract.
• Create with client-known ID: use PUT, or POST with a unique constraint on the ID.
• Create with server-generated ID: use POST with an Idempotency-Key header.
• Message queue consumer: dedup table or outbox pattern, always.
• Payment, order, or email: idempotency keys are non-negotiable.

Conclusion

Idempotency is not an advanced topic you learn once you hit scale. It is a baseline requirement for any API that has retry logic, which is every API. Get the HTTP methods right, add idempotency keys to your POST endpoints that have real-world side effects, use database constraints where you can, and make your queue consumers tolerate replays.

The patterns are not complicated. What matters is being deliberate about applying them before your first duplicate charge, not after.

References

• RFC 9110: HTTP Semantics - Idempotent Methods - Official HTTP specification defining which methods are idempotent
• MDN Web Docs: Idempotent - Accessible explanation of HTTP method idempotency
• Stripe API Documentation: Idempotent Requests - Canonical example of idempotency keys in a payment API
• Stripe Engineering: Designing Robust and Predictable APIs - Deep dive into Stripe’s internal implementation
• AWS Lambda Powertools: Idempotency Utility - Production library for Lambda with DynamoDB backing
• AWS SQS FIFO: Exactly-Once Processing - AWS documentation on deduplication
• IETF Draft: The Idempotency-Key HTTP Header Field - Emerging standard for the header
• PostgreSQL Documentation: INSERT ON CONFLICT - Upsert syntax for idempotent inserts
• DynamoDB Conditional Writes - Condition expressions for idempotent writes
• Apache Kafka: Semantics and Idempotent Producer - Limits of exactly-once semantics
• The Two Generals Problem - Why exactly-once delivery is impossible
• Square Developer Docs: Idempotency - Alternative perspective from another payments provider