Every trade generates a tick: a price, a volume, a timestamp. An active stock might generate thousands of ticks per second. Distributing that data to thousands of subscribers simultaneously is its own problem.

What Tick Data Looks Like#

A tick is small: instrument ID, price, quantity, timestamp. The volume is the problem. During market open or a news event, tick rates spike dramatically. Subscribers range from high-frequency algorithms (latency-sensitive, need every tick) to dashboards (showing “current price,” don’t care about ticks they missed).

These two audiences need different delivery strategies.

Conflation#

Conflation is the market data answer to backpressure. If a subscriber can’t consume ticks as fast as they arrive, you don’t queue up every tick. You keep only the latest. The subscriber gets the most recent state, not the full history of changes.

For a dashboard showing current price, this is fine. You only ever care about the latest quote. For a trading algorithm running a strategy that depends on every price movement, conflation is unacceptable.

graph TD A[Matching Engine] --> B[Tick Publisher] B --> C[High-Frequency Subscriber] B --> D[Conflation Buffer] D --> E{New tick for same instrument?} E -->|Yes| F[Replace Latest Tick] E -->|No| G[Add to Buffer] F --> H[Dashboard Subscriber] G --> H style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

At Oracle#

Oracle telecom systems deal with event streams from network elements. During certain failure conditions, an element would flood the system with status updates, far faster than the monitoring layer could process them. We added a conflation layer: per-element, keep only the latest state update in a map. Consumers pulled from the map on a schedule rather than processing every event. During burst, processing lag dropped from minutes to seconds.

The trade-off was explicit: we were losing intermediate states. For status monitoring this was acceptable. For audit logging it was not, so audit had a separate uncompressed path.

What I’m Learning#

Fan-out strategies and conflation are related problems. Fan-out distributes the same event to many consumers. Conflation reduces what each consumer has to process. You often need both at scale.

Have you implemented conflation, or just let the queue pile up and hoped consumers caught up?