A user uploads a photo. The platform needs: a thumbnail at 150x150, a medium preview at 800x600, the original compressed to 80% quality, EXIF data extracted, content safety scanning, and a perceptual hash for deduplication. None of these should block the upload response. All of them need to happen reliably.

The Fan-Out After Upload#

The upload endpoint does one thing: store the raw file, acknowledge the upload, publish an event. Everything else is async.

POST /photos/upload
  → stores raw file to object storage
  → publishes PhotoUploaded event to Kafka
  → returns 202 Accepted immediately

Downstream workers consume the PhotoUploaded event and perform their specific task. This is a fan-out: one event, multiple independent processors.

The advantage: the upload is fast regardless of how many processing steps follow. Adding a new processing step (say, generating a WebP version) doesn’t touch the upload service at all. You add a new consumer.

The DAG Structure#

Some processing steps have dependencies. You can’t generate a thumbnail from a corrupted file. You need to validate the image first. Content safety scanning might need to run before the image becomes publicly visible. These dependencies form a directed acyclic graph (DAG).

PhotoUploaded
    → Validate image format
        → Generate thumbnail (depends on: validated)
        → Generate preview (depends on: validated)
        → Extract EXIF data (depends on: validated)
        → Content safety scan (depends on: validated)
            → Make publicly visible (depends on: safety scan passed)
graph TD A[PhotoUploaded event] --> B[Image Validator] B --> C[Thumbnail Generator] B --> D[Preview Generator] B --> E[EXIF Extractor] B --> F[Content Safety Scanner] F --> G{Passes?} G -->|Yes| H[Mark as publicly visible] G -->|No| I[Flag for human review] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff

Handling Failures in the DAG#

If thumbnail generation fails, the image is still valid and the original should remain accessible. The retry logic must be per-step, not per-upload. A dead letter queue collects failed processing events. A support engineer can diagnose and reprocess specific steps without re-uploading.

Idempotency is critical: if thumbnail generation runs twice for the same photo (due to a retry), it should produce the same thumbnail, not two conflicting entries in the database.

At Oracle#

We had configuration validation and deployment as a DAG: validate syntax, validate semantics, run integration checks, stage to pre-production, run smoke tests, deploy to production. Initially this was a sequential script. One step failure required restarting from the top. We refactored into a pipeline where each step was an independent job that consumed the output of the previous step and could be restarted individually. Deployment time dropped from 20 minutes (average full run) to 5 minutes when only later stages failed and needed reruns.

What I’m Learning#

Breaking complex processing into independent, idempotent steps lets you retry individual failures rather than starting over. The fan-out pattern keeps the upload path fast. The DAG structure handles inter-step dependencies. These two things together give you a processing pipeline that’s both fast and reliable.

Have you built async processing pipelines where failures in one step didn’t need to undo earlier completed steps?