You want to test two versions of a checkout flow. Half of users should see version A, half should see version B. The same user should always see the same version across sessions. The assignment must happen in milliseconds. And you need to run 50 experiments simultaneously without them interfering with each other.

Deterministic Hashing#

The simplest assignment mechanism: hash the user ID, take the result modulo 100, assign to treatment or control based on the bucket.

int bucket = Math.abs(userId.hashCode()) % 100;
String variant = (bucket < 50) ? "control" : "treatment";

This is deterministic: the same user always gets the same bucket. It’s fast: one hash, one modulo. It requires no database lookup or network call.

The problem with using a single hash space: all experiments share the same bucket boundaries. If experiment A puts users 0-49 in control and experiment B puts users 0-49 in a different control, the two experiments are using the exact same user populations. Any correlation between what experiment A changes and what experiment B measures produces misleading results.

Namespace Isolation#

The fix: mix a namespace identifier into the hash.

String namespace = "checkout_experiment_47";
int bucket = Math.abs((userId + namespace).hashCode()) % 100;

Different namespaces produce different bucket assignments for the same user. User 1001 might be in bucket 72 for experiment 47 (treatment) and bucket 18 for experiment 48 (control). The experiments are now independent.

graph TD A[User 1001 arrives] --> B[Experiment: checkout_v2] A --> C[Experiment: search_ranking_v3] B --> D[Hash user_id + checkout_v2 namespace = bucket 72 = treatment] C --> E[Hash user_id + search_ranking_v3 namespace = bucket 18 = control] D --> F[Show new checkout] E --> G[Show old search ranking] style A fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style B fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style C fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style D fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style E fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style F fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff style G fill:#000000,stroke:#00ff00,stroke-width:2px,color:#fff

Who Gets Included in an Experiment#

Not every user is eligible for every experiment. Experiments might target a specific country, a user tier, or new users only. These eligibility checks happen before the bucket assignment.

Sticky assignment: once a user is assigned to a variant, they should stay in it for the experiment’s duration. Changing someone’s variant mid-experiment corrupts the data: their behavior during the A period is mixed with their behavior during the B period. The hash-based approach is inherently sticky as long as the namespace is stable.

Ramping#

Starting an experiment at 100% traffic is risky. A buggy treatment group causes a bad experience for half of users immediately. Ramp gradually: start with 1% (bucket < 1), expand to 5%, then 10%, then 50%.

// 10% experiment: only users in bucket 0-9 participate
int bucket = Math.abs((userId + namespace).hashCode()) % 100;
if (bucket >= 10) return "not_in_experiment";
return (bucket < 5) ? "control" : "treatment";

At 10% ramp, only 10% of users are in the experiment at all. Within that 10%, half are control and half are treatment. This gives you early signal while limiting exposure.

At Salesforce#

We ran feature experiments using feature flags but without the statistical rigor of proper A/B testing. We’d enable a feature for a percentage of orgs and monitor support ticket volume. The missing piece was a proper control group: we had no systematic way to compare “orgs that got the feature” versus “similar orgs that didn’t.” Several features we thought were successful turned out to have neutral or negative effects that we only discovered after 80% rollout when the trends became undeniable.

What I’m Learning#

Deterministic hashing with namespaces is the foundation of A/B testing at scale. It’s fast, stateless, and produces independent assignments across experiments. The statistical rigor comes from what you do with the assignments, but you have to get the assignment right first.

Have you built experimentation systems from scratch, and what was the hardest part to get right?