Idempotency: Why Retries Need It

Network fails mid-request. Did the payment go through? You don’t know. So you retry.

Now the user is charged twice.

This is why idempotency matters.

What Idempotency Means#

An operation is idempotent if doing it multiple times has the same effect as doing it once.

Idempotent:

• SET balance = 100 (run 10 times, balance is still 100)
• DELETE user WHERE id = 5 (run 10 times, user still deleted once)
• GET /user/123 (reads don’t change state)

Not idempotent:

• UPDATE balance = balance + 100 (run 10 times, balance increases by 1000)
• INSERT INTO orders VALUES (...) (run 10 times, 10 duplicate orders)
• POST /charge without idempotency key (charges customer multiple times)

The problem: networks are unreliable. Requests time out. You don’t know if they succeeded. Retrying non-idempotent operations causes corruption.

The Payment Example#

Client sends payment request. Server processes it, charges card, but response gets lost in network. Client times out, doesn’t know if payment succeeded. Retries.

Server sees new request, charges card again. User charged twice.

Without idempotency, retries cause duplicate charges.

Idempotency Keys#

Client generates unique ID for each logical operation. Server stores it with the result. Retry with same ID? Return cached result instead of re-executing.

@PostMapping("/charge")
public PaymentResponse charge(
    @RequestHeader("Idempotency-Key") String idempotencyKey,
    @RequestBody ChargeRequest request) {
    
    // Check if we've seen this key before
    PaymentResponse cached = paymentCache.get(idempotencyKey);
    if (cached != null) {
        return cached;  // Already processed, return same result
    }
    
    // First time seeing this key
    PaymentResponse response = processPayment(request);
    
    // Store result with idempotency key
    paymentCache.put(idempotencyKey, response);
    
    return response;
}

Client side:

String idempotencyKey = UUID.randomUUID().toString();

HttpRequest request = HttpRequest.newBuilder()
    .uri(URI.create("https://api.example.com/charge"))
    .header("Idempotency-Key", idempotencyKey)
    .POST(bodyPublisher)
    .build();

// Safe to retry with same key
for (int i = 0; i < 3; i++) {
    try {
        HttpResponse<String> response = httpClient.send(request, ...);
        if (response.statusCode() == 200) {
            break;  // Success
        }
    } catch (IOException e) {
        // Retry with same idempotency key
    }
}

Same key = same result. No duplicate charges.

Database Pattern#

Store idempotency key in database with unique constraint. Duplicate key? Transaction fails, but you return the original result.

@Transactional
public PaymentResponse processPayment(String idempotencyKey, ChargeRequest request) {
    try {
        // Insert idempotency record first
        IdempotencyRecord record = new IdempotencyRecord();
        record.setKey(idempotencyKey);
        record.setStatus("PROCESSING");
        idempotencyRepo.save(record);  // Unique constraint on key
        
        // Process payment
        PaymentResponse response = chargeCard(request);
        
        // Update record with result
        record.setStatus("COMPLETED");
        record.setResult(response);
        idempotencyRepo.save(record);
        
        return response;
        
    } catch (DataIntegrityViolationException e) {
        // Duplicate key, already processed
        IdempotencyRecord existing = idempotencyRepo.findByKey(idempotencyKey);
        
        if (existing.getStatus().equals("COMPLETED")) {
            return existing.getResult();  // Return cached result
        } else {
            // Still processing in another thread/server
            throw new ConcurrentModificationException("Request in progress");
        }
    }
}

Database enforces uniqueness. Concurrent retries handled automatically.

Key Expiration#

Can’t store idempotency keys forever. Memory/disk fills up.

Common pattern: Expire after 24 hours. If client retries after expiration, treat as new request. Acceptable for most use cases (who retries after 24 hours?).

// Redis with TTL
redisTemplate.opsForValue().set(
    idempotencyKey, 
    result, 
    Duration.ofHours(24)
);

When to Use Idempotency Keys#

Always:

• Payment processing
• Order creation
• Any financial transaction
• State-changing operations that can’t be undone

Usually:

• Message processing (at-least-once delivery)
• Webhook handlers
• API calls to third parties

Maybe not:

• Internal microservice calls (if you control retry logic)
• Operations that are naturally idempotent (SET, DELETE)

What I’ve Built#

Message processing system at one of the companies I worked with. Messages could be delivered multiple times (at-least-once guarantee). Without idempotency, duplicate messages caused duplicate state changes.

Added idempotency key (message ID). Store processed message IDs in database. Duplicate message? Skip processing, return success. Simple pattern, prevented all duplicate processing issues.

Idempotency isn’t optional in distributed systems. Networks fail. Retries happen. Make your operations safe to retry.

How do you handle idempotency in your APIs?