Definition
Sample ratio mismatch, or SRM, is a statistically unexpected difference between planned experiment allocation proportions and observed group counts. A planned 50:50 split need not be exactly equal, but a sufficiently unusual imbalance can indicate assignment, logging, or filtering problems. This diagnostic examines the comparison population, not the size of a conversion uplift.
Why It Matters
- An apparently winning checkout change can be misleading if one group loses more records before analysis.
- Within a Commerce Intelligence OS, evidence quality should be established before experiment results influence merchandising or agent policy.
- SRM complements holdout design by testing whether the observed sample still follows the intended allocation.
How It Works
- Record the randomization unit, intended allocation, eligible population, and analysis inclusion rules before launch.
- Compare unique assigned-unit counts with expected proportions using an appropriate statistical test and a documented alert threshold. Respect planned allocation changes over time.
- Trace discrepancies through assignment, exposure, event ingestion, and analysis filters. Examine device and time slices to locate the first divergence.
- Resolve the cause before interpreting treatment effects. Preserve diagnostic records; do not randomly discard users merely to force equal group sizes.
Ecommerce Example
Context: Illustrative example: a retailer plans equal assignment across 10,000 shoppers, but the analysis dataset contains 6,000 treatment users and 4,000 control users.
Recommended move: Investigate assignment logs and discover whether a control-only logging failure or a post-assignment filter removed records. A count imbalance alone does not identify the cause.
Why it matters: Treat the conversion comparison as unresolved while that diagnosis proceeds. These hypothetical counts are not an iKawn experiment result.
iKawn Framework
Specify
The iKawn framework stores allocation and inclusion rules alongside the commercial experiment.
Check
Expose sample counts before surfacing an uplift recommendation.
Trace
Connect missing observations to their source stage and owner.
Release
Use reviewed experiment evidence when updating commerce decisions.
Concise Summary
Check whether experiment groups reflect their planned allocation before acting on uplift. An SRM alert is a diagnostic signal requiring investigation, not a correction formula.