Definition
Incrementality holdout design is the practice of reserving an appropriately assigned comparison group that does not receive a specific optional commerce intervention. The purpose is to estimate what changed because of that intervention, rather than counting every order observed after exposure as a success.
Why It Matters
- Customers who receive recommendations or recovery messages may already be likely to buy. Exposure followed by purchase does not by itself establish impact.
- A conversion increase may be offset by discount expense, service cost, cancellations, or returns. The measurement window should match the outcome being evaluated.
- For a Commerce Intelligence OS, the decision is whether an intervention adds enough value to justify continuing it, changing it, or stopping it.
How It Works
- Define eligibility, the optional intervention, the randomization unit, a primary outcome, and the evaluation window before launch. Keep essential service available to both groups.
- Assign treatment and control consistently. Choose customer-level assignment when repeated visits would otherwise expose the same person to conflicting variants.
- Log assignment separately from actual exposure. Check group balance, missing data, cross-group contamination, and concurrent campaigns before interpreting differences.
- Compare outcomes on the predefined assignment basis and report uncertainty. Review return-adjusted contribution after sufficient follow-up rather than declaring victory from early order counts.
Ecommerce Example
Context: Illustrative example: a retailer tests an optional cart reminder with a discount. Equal groups of 1,000 eligible customers produce 120 and 100 orders.
Recommended move: The observed order-rate difference is two percentage points, or 20 additional orders per 1,000 assigned customers. The team still needs uncertainty estimates and a comparison of discount, fulfillment, and return costs.
Why it matters: The example shows why 120 treated-group orders cannot all be credited to the reminder. The numbers are hypothetical and do not establish a statistically reliable result.
iKawn Framework
Specify
Use the iKawn framework to state the intervention and the commercial outcome it should improve.
Assign
Preserve treatment membership and eligibility as explicit commerce context.
Observe
Join exposure, orders, costs, and mature return outcomes.
Decide
Scale only when the evidence supports additional value within the chosen uncertainty and cost limits.
Concise Summary
A holdout estimates additional value by preserving a credible comparison. Stable assignment, a predefined outcome window, and downstream cost measurement make the result more useful than attributed revenue alone.