Definition
A missingness indicator is a feature that records whether another input was absent before imputation. For example, a model can receive both a filled product-weight value and a flag showing that the original weight was missing. The flag preserves an observation about data availability; it does not recover the true missing value.
Why It Matters
- Replacing every missing measurement with a median can make an unknown value look identical to a real measurement at that median.
- A Commerce Intelligence OS needs visibility into whether predictive recommendations rely on measured product evidence or a fallback introduced by preprocessing.
How It Works
- Define which source states count as missing for each feature. Preserve the raw input and distinguish unknown values from confirmed zero values.
- Create availability flags before filling missing values. Fit any learned imputation values only on the training portion of each evaluation split.
- Keep feature columns and preprocessing consistent during training and serving. Test what happens when a feature first becomes missing after deployment.
- Evaluate whether flags improve held-out performance and calibration. Monitor source outages and changing missingness patterns because the model may learn fragile relationships tied to a data collection process.
Ecommerce Example
Context: Illustrative example: one product has a measured weight of 500 grams. Another has no weight, and the training median used to fill it is also 500 grams.
Recommended move: Both numeric inputs become 500, but the missingness flag separates measured evidence from a fallback. Compare models with and without that flag on future-held-out data.
Why it matters: A useful improvement is possible, not guaranteed. An outage that makes many more weights absent can invalidate the learned association; this is a hypothetical modeling scenario.
iKawn Framework
Preserve
The iKawn framework retains the distinction between observed and unavailable evidence.
Prepare
Version indicators and learned preprocessing values together.
Evaluate
Measure whether the added availability information improves relevant decisions.
Monitor
Track missingness changes alongside predictive performance.
Concise Summary
Missingness indicators retain data-availability information through imputation. They require consistent preprocessing and validation because missing-data patterns can change.