Definition
Prediction interval coverage is the share of evaluated outcomes that fall inside their previously issued forecast intervals. An 80% prediction interval is intended to cover the future observation about 80% of the time across comparable repeated forecasts under the model assumptions. It describes uncertainty around an outcome, rather than a guarantee for one SKU or week.
Why It Matters
- A precise point forecast can hide uncertainty that materially changes stock and capacity decisions.
- Consistently narrow ranges can encourage planners to treat uncertain demand as settled fact.
- Predictive commerce needs both coverage and interval width: an extremely wide range may cover outcomes but provide little planning value.
How It Works
- Define the forecast target, time horizon, probability level, and evaluation population. Keep demand and observed sales distinct where stockouts censor purchases.
- Save intervals when issued and compare them with later outcomes over rolling forecast origins, using only information available at each origin.
- Report empirical coverage, interval width, and an interval scoring rule that accounts for width and misses. Inspect categories and horizons separately.
- Investigate persistent undercoverage or overcoverage and evaluate revised models on subsequent held-out periods. Retain sample counts and dependence between forecast errors when judging uncertainty.
Ecommerce Example
Context: Illustrative example: a planner stores 100 one-week-ahead 80% demand intervals. Only 62 later outcomes fall inside their respective ranges.
Recommended move: Report 62% observed coverage, inspect misses by product group, and compare interval widths before changing the uncertainty model.
Why it matters: This is a diagnostic gap relative to the nominal 80%, not proof of a specific cause or a promise that every future group will achieve exactly 80%.
iKawn Framework
Preserve
The iKawn framework retains forecast ranges and their issue times.
Observe
Join later outcomes to the matching target and horizon.
Assess
Show range coverage beside sharpness and operational constraints.
Adapt
Review uncertainty estimates before agents use them in planning.
Concise Summary
Evaluate forecast ranges against later outcomes. Useful uncertainty estimates need credible coverage and informative width, with the target, horizon, and sample size visible.