Definition
CUPED, or Controlled-experiment Using Pre-Experiment Data, adjusts an experiment outcome using a correlated measurement taken before treatment. In a simple form, the adjusted outcome is Y - theta * (X - mean(X)), where Y is the experiment metric and X is the pre-experiment covariate. The aim is reduced estimator variance without changing the intended treatment effect.
Why It Matters
- Returning shoppers differ in their usual purchase activity. That variation can make small merchandising effects difficult to estimate.
- Pre-existing differences can be measured before an intervention and accounted for in a planned analysis.
- A Commerce Intelligence OS needs the raw outcome and adjustment method together so a commercial reviewer can understand what changed.
How It Works
- Predefine the experiment unit, outcome, pre-period, and covariates. Use information unaffected by treatment and preserve randomized assignment.
- Estimate the adjustment coefficient using an appropriate planned procedure; in the simple case theta is covariance of Y and X divided by variance of X. Handle zero variance and missing histories explicitly.
- Compare treatment and control using adjusted outcomes and suitable uncertainty estimates. Account for clustering when the assignment unit requires it.
- Retain raw and adjusted estimates, sample coverage, and observed variance reduction. Check assignment quality and mature return outcomes before deciding whether a rollout is justified.
Ecommerce Example
Context: Illustrative example: a merchant tests a reorder interface among eligible existing customers and uses their pre-test purchasing activity as a covariate.
Recommended move: The analyst checks whether that history predicts the test-period metric and reports both unadjusted and adjusted differences. New customers need a predefined missing-history treatment.
Why it matters: The adjustment may improve precision, but it cannot manufacture an intervention benefit. No numerical gain or iKawn customer outcome is implied.
iKawn Framework
Plan
The iKawn framework records the commercial hypothesis and analysis choices.
Contextualize
Keep pre-treatment history distinct from treatment outcomes.
Estimate
Present adjusted effects alongside raw observations and uncertainty.
Decide
Use trustworthy evidence and commercial relevance to guide rollout.
Concise Summary
CUPED can reduce experiment noise using pre-treatment information. It complements randomized design and does not repair biased assignment or guarantee a positive result.