Definition
Quantile loss, also called pinball loss, evaluates a forecast of a selected demand quantile. For quantile level q, an underprediction of d units costs q times d; an overprediction of d units costs (1 minus q) times d. Lower average loss is better when comparing forecasts on the same observations and quantile level.
Why It Matters
- A replenishment team may need an upper demand estimate rather than an average because shortages and excess stock have different consequences.
- An evaluation should match the forecast target. A median and a 90th-percentile forecast answer different questions.
- For a Commerce Intelligence OS, the scoring rule makes the planning preference explicit before an agent compares candidate forecasts.
How It Works
- Choose the target, forecast horizon, quantile level, and evaluation sample before comparing models. Keep unavailable sales periods visible when demand is censored.
- Save each forecast at its issue time. Calculate the asymmetric loss against the later observed target and average it using documented weights.
- Compare a simple historical benchmark and candidate models on identical rolling evaluation periods. Inspect SKU groups separately so high-volume products do not silently dominate the result.
- Review calibration and operational consequences alongside loss. Selecting a quantile does not by itself determine an order quantity when lead times, current stock, pack sizes, and costs also matter.
Ecommerce Example
Context: Illustrative example: a 90th-percentile forecast is 100 units. Actual demand of 120 produces loss of 0.9 times 20, or 18.
Recommended move: If actual demand were 80 instead, the loss would be 0.1 times 20, or 2. The scoring rule penalizes an equally sized shortage-side error more heavily.
Why it matters: Use many held-out cases before ranking models. These hypothetical calculations demonstrate the score, not an iKawn customer result.
iKawn Framework
Specify
The iKawn framework attaches a forecast quantile and horizon to the planning question.
Score
Evaluate saved predictions against later commerce evidence.
Compare
Show benchmark performance and category-level differences.
Decide
Combine forecast quality with inventory constraints before action.
Concise Summary
Quantile loss evaluates a chosen part of the demand distribution. Compare like-for-like forecasts and keep forecast scoring separate from the full replenishment decision.