Definition
Simpson paradox occurs when a relationship observed within groups reverses after those groups are combined. In ecommerce, a change in the mix of devices, customer types, or channels can make an aggregate conversion rate fall even while each segment improves. Aggregate rates weight segments by their denominators, so changing weights can change the apparent story.
Why It Matters
- A single headline rate can prompt a team to roll back a change despite improvements within every relevant segment.
- The reverse can also happen: a favorable mix shift can hide weaker segment performance.
- A Commerce Intelligence OS should expose denominators and composition alongside rate changes before recommending commercial action.
How It Works
- Define the comparison population, outcome, and denominator. Confirm that tracking and eligibility rules remained comparable.
- Inspect plausible segments chosen for the business question, showing their counts and rates in both periods. Avoid searching arbitrarily for a preferred conclusion.
- Calculate a descriptive comparison using fixed segment weights to isolate the arithmetic role of composition. Present it beside actual aggregate performance.
- Choose a causal analysis or experiment appropriate to the decision before attributing the change to an intervention. Segment adjustment alone does not establish causation, and the causal role of the segment variable matters.
Ecommerce Example
Context: Illustrative example: period A has 90 orders from 900 desktop sessions and 2 from 100 mobile sessions: 9.2% overall. Period B has 11 from 100 desktop sessions and 27 from 900 mobile sessions: 3.8% overall.
Recommended move: Desktop conversion improved from 10% to 11%, and mobile from 2% to 3%. The shift toward mobile explains the reversal in the combined rate. At the original 90:10 session mix, period B would be 10.2%.
Why it matters: These hypothetical calculations describe composition, not proof that a website change caused either segment improvement.
iKawn Framework
Expose
The iKawn framework keeps segment counts next to performance rates.
Compare
Separate observed aggregate changes from within-segment changes.
Explain
Show how changing weights contribute to the headline result.
Decide
Match causal evidence and the relevant population to the commercial question.
Concise Summary
When aggregate and segment trends disagree, inspect their weights. A composition-aware explanation prevents arithmetic reversals from becoming unsupported causal claims.