A model update ships. Fraud scores change. Conversion rates drop. You need to roll back to the previous model in minutes, not hours. If your model deployment is treated differently from code deployment, rollback is painful. If models are versioned and deployed with the same tooling as code, rollback is a one-command operation.

What Model Versioning Means#

A model is a file (or set of files) produced by a training run. Versioning means: every trained model is stored with a unique identifier, its training metadata (what data it was trained on, what hyperparameters were used, when it was trained), and its performance metrics on a validation set.

Model Registry:
  fraud_detection/v1: trained 2026-04-01, AUC 0.923, validation set apr-2026
  fraud_detection/v2: trained 2026-05-15, AUC 0.941, validation set may-2026
  fraud_detection/v3: trained 2026-06-10, AUC 0.938, validation set jun-2026 (current)

The serving layer doesn’t load model files directly from the filesystem. It requests a model by name and version from a model registry, which returns the file location. Changing which version is active is a registry update, not a file replacement.

Rollback#

Rolling back from v3 to v2: update the registry entry for fraud_detection/current to point to v2. The serving layer either polls the registry periodically or watches for updates via a mechanism like etcd watches. Within seconds, all serving instances load v2 and v3 predictions stop.

The key: both models are already stored. Rollback doesn’t require retraining or file transfers. It’s a metadata change.

graph TD A[Model Registry] --> B[current: fraud_detection v3] C[Problem detected with v3] --> D[Rollback: update current to v2] D --> A A --> E[Serving instances: poll registry] E --> F[Detect version change: current is now v2] F --> G[Load v2 from storage] G --> H[Serve v2 predictions within 30 seconds] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Canary Model Deployment#

Before rolling a new model out to all traffic, send a small percentage to the new model and compare outcomes against the current model. This is model canary release: route 5% of predictions to v3, keep 95% on v2. Watch the divergence in scores and any downstream business metrics. If v3 looks good on the 5%, gradually shift more traffic.

This requires the serving layer to support routing a percentage of requests to a specific model version. The assignment can be based on request hash (same transaction always goes to the same model version during the test), or fully random per-request.

Training-Serving Skew#

A common model deployment failure: the model was trained on data preprocessed one way, but the serving layer preprocesses features a different way. The model receives features in an unexpected format and produces garbage predictions without any obvious error.

Preventing this: the feature preprocessing logic (normalization, encoding, imputation) should be versioned alongside the model, deployed together, and tested together. Keeping them separate invites divergence.

At Salesforce#

We had business logic engines (essentially rule-based models) that needed versioning. An update to the rule engine that changed how validation worked couldn’t easily be rolled back because the rule engine version wasn’t tracked separately from the application version. A bad rule engine update required a full application rollback, taking 20 minutes. We eventually extracted the rule definitions into a versioned configuration and separated them from the application deployment. Rollback went from 20 minutes to under 2 minutes. Same principle as model versioning: separate the “logic” from the “runtime.”

What I’m Learning#

Model versioning is not a nice-to-have. The first time a model update causes a production incident and rollback takes an hour, it becomes obvious. The registry pattern, decoupling model identity from model files, is what makes fast rollback possible.

Have you built systems where the “model” or “rules” needed to be versioned independently of the application, and how did you implement rollback?