Getting code from a merged PR to production involves more than copying files to servers. A deployment must: roll out gradually, monitor for regressions, pause or roll back automatically on failure signals, and coordinate across dozens of services that depend on each other. Deployment orchestration is the system that makes this happen reliably without human intervention on the critical path.

The Deployment State Machine#

Each deployment is a state machine: pending, running, paused, succeeded, rolled_back. Transitions are driven by health signals and manual overrides. An orchestrator polls health metrics and drives the state machine forward or backward.

Start: deploy to 1% of instances. Wait 5 minutes. Check error rate, latency p99, and business metrics against pre-deployment baseline. If all green, advance to 10%. Continue through 25%, 50%, 100%. If any step fails health checks, pause and alert. If the metric exceeds a rollback threshold, automatically roll back.

Canary vs Rolling vs Blue-Green#

Canary: a small percentage of traffic routes to the new version while the rest stays on old. Uses a load balancer weight or feature flag to split traffic. Gradual ramp reduces blast radius.

Rolling: replace instances one at a time. Each instance is drained (stop receiving new requests, finish in-flight), updated, and brought back. At 10% completion, 10% of instances run the new version. No traffic splitting needed — the load balancer naturally distributes across available instances.

Blue-green: maintain two identical environments. Switch all traffic at once. Instantaneous cutover, easy rollback (switch back). Doubles infrastructure cost.

graph TD A[Deploy v2 triggered] --> B[Stage: 1% of instances, 5 min soak] B --> C{Health check: error rate, p99 latency} C --> |Pass| D[Stage: 10%, 10 min soak] D --> E{Health check} E --> |Pass| F[Stage: 50%, 15 min soak] F --> G{Health check} G --> |Pass| H[Stage: 100%, deployment complete] C --> |Fail| I[Auto-rollback: revert 1% to v1] E --> |Fail| I G --> |Fail| I style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style H fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style I fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Dependency Ordering#

Service B depends on Service A. Deploying both simultaneously risks deploying B’s new code before A’s API changes are available. Deployment orchestration must respect dependency graphs: deploy dependencies first, dependents after. This is a topological sort of the service dependency graph.

For backward compatibility: deploy A’s new version first (which is backward compatible with B’s old version), then deploy B. This requires planning breaking API changes as two-phase deployments: add-then-remove rather than replace.

Rollback Triggers#

Automatic rollback fires on: error rate increasing by more than X% over baseline, p99 latency exceeding Y ms, key business metrics (orders, signups) dropping by Z% from the pre-deployment baseline. These thresholds must be tuned per service — a payment service rollback threshold is tighter than a recommendation service.

At Salesforce#

Salesforce’s release orchestration for major platform releases (Spring/Summer/Winter releases) deployed changes to 100+ microservices across 50+ customer orgs. The orchestrator tracked service dependency order, deployed in waves, and measured error rates per org after each wave. If error rates exceeded 0.5% in any org after a wave, the deployment paused automatically and required human sign-off to proceed. This caught 3 deployment issues in the last major release cycle before they reached all orgs.

What I’m Learning#

Deployment orchestration is fundamentally about reducing blast radius and automating the rollback decision. The health check thresholds and soak times are the hard part to tune: too tight and you false-positive on noise; too loose and you let bad code reach 100% before triggering rollback.

What metric has been most reliable as an early warning signal in your deployment pipelines?