Definition
Keyset pagination advances through an ordered dataset using the last retrieved key rather than a numeric row offset. A composite key can combine a timestamp with a unique identifier to break ties. It defines how a scan moves forward; it does not by itself provide a consistent snapshot or capture all later changes.
Why It Matters
- Order and return imports often span many requests. An interrupted job needs a precise position from which to resume.
- For a Commerce Intelligence OS, an incomplete import can distort retained-revenue and return metrics even when each individual request succeeds. Scan completeness needs its own evidence.
How It Works
- Choose a deterministic total order and a supported seek predicate. Prefer immutable scan keys where possible, and use a unique tie-breaker when the main sort value repeats.
- Fetch records strictly after the saved key in that same order. Apply the batch durably before advancing the checkpoint; make reapplication safe if a crash occurs between those steps.
- Define consistency separately through a source snapshot or another documented extraction contract. An upper bound alone cannot protect against every late insert, update, or deletion.
- Reconcile source counts or control totals where meaningful. Use a change feed, bounded overlap with deduplication, or periodic reconciliation for changes outside the completed scan.
Ecommerce Example
Context: Illustrative example: several return records share the same creation timestamp, and a batch ends at record ID 104 within that timestamp.
Recommended move: Resume after the composite timestamp-and-ID key, allowing IDs 105 and 106 at the same timestamp to be included.
Why it matters: A later backdated record can still fall behind the cursor. A separate change-capture or reconciliation process must handle it; the example does not promise exactly-once delivery.
iKawn Framework
Order
The iKawn ontology framework retains source identity and extraction keys.
Checkpoint
Record progress only after durable batch application.
Recover
Make replayed batches safe to apply.
Reconcile
Verify coverage before downstream decisions use the imported dataset.
Concise Summary
Keyset pagination provides a resumable ordered scan. Completeness also requires a consistency contract, replay-safe writes, and a strategy for late changes.