Last post: exactly-once delivery is impossible across system boundaries. So what actually works?

At-least-once delivery plus idempotent consumers. Kafka keeps delivering until you commit. Messages might repeat, but they won’t get lost. Your consumer handles duplicates gracefully.

Database-Level Deduplication#

The trick: store the message ID in the same transaction as your business logic.

@Transactional
public void processOrder(OrderEvent event) {
    String messageId = event.getMessageId();

    // Unique constraint on message_id prevents duplicates
    try {
        processedMessageRepo.save(new ProcessedMessage(messageId));
    } catch (DataIntegrityViolationException e) {
        log.info("Duplicate message {}, skipping", messageId);
        return;
    }

    orderRepo.save(new Order(event.getOrderId(), event.getItems()));
}

Duplicate message? Constraint violation. Skip. Transaction rolls back. Clean.

This is the Transactional Outbox in reverse. There, we ensured events publish exactly once. Here, we ensure they’re processed exactly once.

The External Service Problem#

Database writes are easy to make idempotent. External calls are harder. You can’t roll back a sent email.

Three options. If the external service accepts an idempotency key, pass the message ID. If not, track the call status in your database, in the same transaction as the business write. Store an “email_sent” flag alongside the order, check it before sending. Or for low-stakes operations, just accept occasional duplicates and move on. Two confirmation emails is annoying but not catastrophic.

At Oracle, this pattern cut our duplicate notification processing by 50% on the NSSF project. We stopped fighting Kafka’s delivery semantics and started making consumers safe.

graph TD M[Message Arrives] --> C{Already Processed?} C -->|Yes| S[Skip] C -->|No| P[Process + Save ID] P --> D{Success?} D -->|Yes| K[Commit Offset] D -->|No| R[Retry Later] style M fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style S fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style P fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style K fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style R fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff

What I’m Learning#

“Exactly-once” is marketing for “we handle the hard part, you handle the rest.” Kafka’s exactly-once is real, but it ends at Kafka’s boundary. Your job is making processing safe so duplicates don’t matter.

Every message system teaches this lesson eventually. Kafka, RabbitMQ, SQS. Networks fail. Retries happen. Make your consumers safe.

What message system do you use? How do you handle duplicate processing?