You have experiment assignments: which user got which variant. You have metric events: user completed checkout, user added to cart, user bounced. To compute experiment results, you need to join assignments to events. At millions of events per day across dozens of experiments, this join is the core engineering challenge of an experimentation platform.

The Data Model#

Three streams of data:

Assignment events: user 1001 was assigned to treatment in experiment “checkout_v2” at 14:32:00. These are written at the moment of assignment.

Metric events: user 1001 completed a purchase at 14:45:00. These are written at the moment the metric occurs.

To compute conversion rate per variant, join on user ID: find all metric events for users in treatment, find all metric events for users in control, compare.

SELECT
    a.variant,
    COUNT(DISTINCT a.user_id) AS users,
    COUNT(m.user_id) AS conversions,
    COUNT(m.user_id) * 1.0 / COUNT(DISTINCT a.user_id) AS conversion_rate
FROM experiment_assignments a
LEFT JOIN metric_events m
    ON a.user_id = m.user_id
    AND m.event_type = 'purchase'
    AND m.occurred_at BETWEEN a.assigned_at AND a.assigned_at + INTERVAL 7 DAY
WHERE a.experiment_id = 'checkout_v2'
GROUP BY a.variant;

The Scale Problem#

This join on raw event tables works at small scale. At millions of users and hundreds of millions of events, joining on user ID across both tables produces a query that runs for hours. You can’t wait hours to see experiment results.

The solution: pre-aggregate. Instead of joining raw events, maintain running aggregates per experiment per variant: total users assigned, total conversions, total revenue, updated continuously as new events arrive.

graph TD A[Assignment event: user 1001, checkout_v2, treatment] --> B[Assignment table] C[Metric event: user 1001, purchase, $49] --> D[Event stream: Kafka] B --> E[Join service: user in which experiment variants?] D --> E E --> F[Update aggregate: checkout_v2 treatment: +1 conversion, +$49] F --> G[Dashboard reads aggregates: near real-time] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Delayed Attribution#

A user is assigned to a variant at 10am. They complete a purchase at 11pm the same day. The metric event needs to be attributed to the correct experiment variant even hours after assignment.

Two approaches: at event time, look up the user’s current experiment assignments and attribute the event immediately. Or store raw events and assignments separately, and run attribution as a batch join nightly.

Real-time attribution is accurate but requires looking up assignments for every event, adding latency to the event processing pipeline. Batch attribution is simple but means results are only updated daily.

At Oracle#

We tracked which configuration version was active when a service incident occurred. If an incident happened, we needed to know: which config change preceded it? We kept an append-only log of config changes with timestamps and joined it against incident timestamps to find “config active at time of incident.” It was manual and ad-hoc. A proper experiment tracking system would have made this a structured query. The data was there; the tooling wasn’t.

What I’m Learning#

Experiment metric pipelines are essentially a join problem between two high-volume streams. The engineering choices are between real-time joins (accurate, complex) and batch joins (simple, delayed). Most experimentation platforms do both: real-time for watching experiments, batch for final analysis.

Have you built systems that needed to correlate events across time with some “context” that was set earlier (like experiment assignment or configuration state)?