You have a statistically significant result: the new checkout flow increases conversion by 2%. Now what? You don’t immediately flip it on for everyone. You ramp it: gradually increase the percentage of traffic seeing the new experience, watching for problems that didn’t show up in the experiment.

Why Ramp After a Successful Experiment#

The experiment ran at 10% traffic for two weeks. It didn’t show problems. But some issues only appear at scale:

Infrastructure load: 10% of traffic might not saturate the new checkout service. At 100%, it might. Infrastructure problems that didn’t appear at 10% appear at 50%.

Edge cases: rare user paths (unusual account states, rare device types, old browsers) might not have been represented in 10% of users. At full rollout, all edge cases appear.

Time-based effects: a two-week experiment might miss monthly subscription interactions, quarterly billing cycles, or seasonal patterns.

Ramping exposes these problems in sequence: if something breaks at 25%, you’ve limited blast radius to 25%, not 100%.

Automated Guardrails#

Manual monitoring during ramp doesn’t scale across dozens of experiments. Automated guardrails watch key metrics and halt the ramp if they degrade.

Define guardrail metrics before starting: error rate, p99 latency, payment failure rate. Set thresholds: if error rate increases more than 1% above baseline, stop the ramp and alert.

The ramp schedule might be: 5% for 1 day, 10% for 1 day, 25% for 1 day, 50% for 1 day, 100%. At each step, guardrail metrics are checked. If any trigger, the ramp stops and the experiment team is paged.

graph TD A[Experiment: checkout_v2, 2% uplift] --> B[Ramp to 5%: 1 day] B --> C{Guardrails OK?} C -->|Yes| D[Ramp to 25%: 1 day] C -->|No| E[Stop ramp, alert team, investigate] D --> F{Guardrails OK?} F -->|Yes| G[Ramp to 100%] F -->|No| E G --> H[Experiment complete: ship] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#ff0000,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Holdback Groups#

After a feature ships at 100%, you might still want to measure its long-term effect. Holdback groups: keep a small percentage (1-5%) on the old experience permanently. Compare their metrics against the full 95-99% on the new experience to validate that the uplift persists over time and doesn’t decay.

Holdback groups are controversial: keeping some users on a worse experience (if the experiment was positive) for measurement reasons requires justification. The standard answer: a 1% holdback for 30 days is a small cost for knowing whether the positive result was real.

Experiment Cleanup#

A shipped feature that was controlled by an experiment flag needs cleanup: remove the flag code, remove the control path, delete the flag from the config system. Accumulated experiment flags in code create technical debt. Feature flags are long-lived; experiment flags are short-lived and need to be cleaned up aggressively.

At Salesforce#

We had no formal ramp process. Features that tested well in sandbox would sometimes cause issues in production at full scale. After one incident where a “validated” config change caused 20% of API calls to fail for a subset of orgs, we added a staged rollout: push to 10% of orgs, watch for 2 hours, then push to all. It was manual, not automated, but it caught two more incidents before they became full outages. Automation came later. The manual process was better than nothing.

What I’m Learning#

A successful experiment result tells you the treatment is better in expectation. Ramp with guardrails tells you it’s safe in practice. Both pieces are needed. The guardrail automation is what allows teams to ship quickly without requiring someone to manually watch dashboards during every ramp.

Have you had a feature that tested well but caused problems during ramp-up, and what did the guardrails miss?