Definition
A paired forecast comparison aligns two model predictions with the same SKU, location, forecast origin, horizon, and realized target. It computes each model loss for that observation and then the difference between losses. Pairing controls which cases are compared; statistical uncertainty still depends on how those cases are related.
Why It Matters
- One model can look better simply because it failed to produce forecasts for the hardest products. Comparing unmatched dashboard averages hides this selection.
- The Commerce Intelligence OS framework needs reproducible evidence about which planning decisions improved and which products remain unsupported.
How It Works
- Predefine the eligible cases, loss function, direction of improvement, and business weights. Preserve forecast issue times to prevent later predictions entering an earlier decision comparison.
- Join both predictions to the same realized outcomes and compute paired loss differences. Report excluded and missing pairs separately rather than silently discarding model failures.
- Summarize the mean or another predefined measure of differences and inspect product groups. Fix the denominator and weights across the comparison.
- Assess uncertainty with a procedure appropriate to dependence across time and related SKUs. A simple paired t-test assumes independent pair differences; overlapping forecast horizons can violate that assumption.
Ecommerce Example
Context: Illustrative example: for three matched planning cases, model A has absolute errors of 8, 4, and 6 units; model B has 6, 5, and 3.
Recommended move: The A-minus-B differences are 2, -1, and 3 units, averaging about 1.33 units in favor of B on these cases.
Why it matters: Three cases are not enough to establish a reliable general improvement. Retain the per-case evidence, inspect coverage, and evaluate across representative periods before changing a buying rule.
iKawn Framework
Match
The iKawn framework identifies the same commercial decision across model versions.
Compare
Retain losses and their paired differences.
Qualify
Show coverage gaps and dependence-aware uncertainty.
Review
Connect the observed differences to merchant priorities before rollout.
Concise Summary
Compare models on the same decision cases and expose missing coverage. Pairing clarifies differences, while dependence-aware analysis determines how confidently they generalize.