You have a new fraud detection model. It performs better in offline evaluation: higher precision, higher recall on historical data. But offline metrics don’t always translate to production. A model can look great on historical data and behave unexpectedly when it sees live traffic with its latency constraints, real-time features, and edge cases that didn’t appear in the training set.

Shadow mode lets you run the new model in production without affecting users.

How Shadow Mode Works#

In shadow mode, every incoming request is sent to both the production model and the shadow (candidate) model simultaneously. The production model’s response is returned to the user. The shadow model’s response is logged but ignored.

Request arrives → Production model → Response to user (10ms)
                → Shadow model   → Response logged only (15ms)

Both run in parallel. User never sees shadow model's decision.

After running in shadow for a week, you compare the shadow model’s decisions against the production model’s decisions. Where do they disagree? When the shadow model says “high risk” and the production model says “low risk,” which one is correct? Analysts review the divergence cases to validate whether the new model is actually better.

graph TD A[Incoming transaction] --> B[Feature Assembly] B --> C[Production Model v2: decision used] B --> D[Shadow Model v3: decision logged only] C --> E[Allow or block transaction] D --> F[Log prediction for offline analysis] F --> G[Compare shadow vs production decisions] G --> H[Review divergences: which model was right?] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

What Shadow Mode Validates#

Latency: does the new model meet the latency budget? If production model takes 12ms and shadow takes 80ms, it doesn’t matter how accurate it is. Shadow mode surfaces this immediately.

Feature availability: new models often use features that the online feature store doesn’t yet serve efficiently. Shadow mode reveals which features are missing or slow to fetch in real-time, before those gaps affect production.

Decision distribution: what percentage of transactions does each model flag? A new model that flags 40% of transactions as high risk (vs 5% for production) has a problem, even if its offline precision metrics looked good.

Shadow vs A/B Test#

Shadow mode is not A/B testing. In shadow mode, the new model has no impact on users. It’s pure observation. A/B testing exposes real users to the new model’s decisions.

Use shadow mode to validate technical correctness and production behavior. Use A/B testing to validate business impact (does blocking more fraud with the new model reduce fraudulent charges without blocking too many legitimate users?). Shadow comes first.

At Oracle#

We had a new configuration validation engine we wanted to roll out. The old engine was conservative and approved almost everything. The new engine was stricter. Before switching production traffic, we ran both engines in parallel for a month, logging divergences. We found the new engine rejected 3% of configs that the old one accepted. Manual review of those divergences revealed that 2% were genuinely invalid configs that the old engine was silently allowing, and 1% were false positives in the new engine. We fixed the false positives before switching. Without shadow mode, we would have rejected valid customer configurations on day one.

What I’m Learning#

Shadow mode is a risk-free way to observe a new system’s behavior in production conditions. The cost is running the new system in parallel (compute cost) and building the logging and comparison infrastructure. For high-stakes decisions (fraud, content moderation, credit risk), that cost is always worth it.

Have you used shadow mode or traffic mirroring to validate a new component before switching traffic to it?