A model trained offline achieves 94% accuracy. In production, accuracy is 78%. The model didn’t change. The data did, in subtle ways you didn’t notice. This is training-serving skew: the distribution of features the model sees during training doesn’t match what it sees during serving.

Where Skew Comes From#

Feature preprocessing differences: the training pipeline normalized “amount” by dividing by the maximum amount in the training dataset. The serving pipeline didn’t. The model gets feature values in a completely different scale during serving.

Missing values: the training data had no null values because the data pipeline cleaned them before training. The serving pipeline receives raw data from upstream services that sometimes send null. Nobody defined what to do with null values in the serving path.

Feature computation lag: “user’s 30-day average transaction amount” was computed from a clean historical batch in training. In serving, it’s computed from a live database that includes in-flight transactions. The training and serving versions of this feature are computed from different data windows.

Temporal drift: the model was trained on data from six months ago. User behavior has shifted. The model’s learned patterns no longer match current reality.

graph TD A[Training Pipeline] --> B[Data cleaning and normalization] B --> C[Feature computation: batch, offline] C --> D[Model trained on clean features] E[Serving Pipeline] --> F[Raw data from upstream services] F --> G[Feature computation: real-time, different logic] G --> H[Model receives different feature distribution] D --> I[Model expects training distribution] H --> J[Skew: model sees unexpected values] I --> J J --> K[Degraded prediction quality] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style J fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style K fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff

Detecting Skew#

Log features at serving time. Compare the distribution of serving features against the distribution of training features for each feature dimension. If the mean, standard deviation, or percentile distribution of a feature has shifted significantly, something is wrong.

This requires feature logging infrastructure: every prediction request, log the features alongside the prediction. Run distribution comparison jobs periodically (daily or more often for high-stakes models). Alert when a feature’s distribution shifts beyond a threshold.

The Feature Store Solution#

The root cause is usually that training and serving compute features differently. The fix: use the same feature computation code for both. A feature store with a unified feature definition layer ensures that “user_30d_avg_transaction” is computed identically whether you’re training a model or serving predictions. Training reads from the offline feature store; serving reads from the online feature store, but both use the same feature definition.

Training pipelines that use pre-computed features from the feature store automatically use the same features as the serving layer. Skew from computation differences is eliminated structurally.

At Oracle#

We had configuration scoring models (rule-based, not ML) that were “trained” on example configurations and then applied to production configs. The training examples were idealized, manually crafted configs. Production configs had quirks: fields set to sentinel values (0 or -1 for “not configured”), encoding differences between system vendors. When the scoring engine saw these sentinel values, it produced nonsensical scores. We fixed it by running production traffic through the scoring engine in shadow mode and reviewing every case where the score was outside the expected range, building a catalog of production edge cases to handle.

What I’m Learning#

Training-serving skew is almost always a symptom of two separate code paths computing the same thing differently. The best prevention is a single feature computation library shared between training and serving, not two implementations that try to stay in sync.

Have you encountered training-serving skew in production, and how did you detect it?