2.4
ShelfTrace
Real shelves in real stores, where every product box records where it sits, what it is, whether it is real, who decided, and what it sold.
Status: Private, with partnership access. ShelfTrace is not released and will not be open-sourced; it trains and evaluates Shelf-1.
Most retail vision data is a photo and a box. ShelfTrace is a record of decisions. Each product box on a shelf photo is tied to a bay on the store's floor plan, a product in a catalogue of more than 30,000 items, a validity class, the full history of who judged it and what alternatives they were shown, and the daily till sales of that product in that store. Today it holds a complete grocery store in Southeast Asia: every bay photographed, about 41,500 product boxes, 8,692 distinct products on shelf, 31,660 boxes with verifier records, and 1.4 million product-days of sales. It grows by about 25,000 boxes with every store walked. ShelfTrace is not public; it is the training and evaluation set behind Shelf-1.
1. Why another shelf dataset
Public shelf datasets answer one question: where are the products? SKU-110K11. SKU-110K: Goldman et al., Precise Detection in Densely Packed Scenes, CVPR 2019. Code and data: SKU110K_CVPR19. does that well, with one class, "product", for every object. Product recognition sets answer a second: which product is this crop? Neither tells you whether the answer was right, who said so, what else it could have been, or whether it mattered to the store.
Those are the questions a retailer pays to answer, and the ones a model needs to learn when to trust itself. ShelfTrace is built from a working system in which associates photograph shelves, a model proposes, a verifier and people decide, and the tills report what sold. The dataset is the exhaust of that system, kept on purpose.
2. What one record holds
| Layer | Fields |
|---|---|
| Where | Store, department, bay on the CAD floor plan, plan coordinates, shelf row and slot, photo version, capture time, the four-corner shelf polygon the associate drew. |
| What | Box in pixels and normalised, detector confidence, product id, three-level category, text read from the pack, pack type (pouch, box, bottle, jar, can). |
| Is it real | Valid product, or one of seven non-product and hard classes: price tag or label, shelf edge or fixture, empty shelf, partial or occluded, several products in one box, too small or blurry, other. |
| Who decided | Every judgement in order: the model's own pick and scores, the verifier's probabilities, the ranked candidates a person was shown, the action they took, their role, the time taken, and the exact model and index versions behind the guess. |
| What it sold | Daily till revenue for that product in that store, joined to the bay through the product id. |
Table 1. Annotation layers.
2.1 An example record
Illustrative; identifiers and values are changed.
{
"image": { "store": "s-01", "bay": "G3·A4", "version": 3, "captured_at": "2026-09-09",
"plan_xy": [412.5, 118.0], "shelf_polygon": [[38,100],[1844,95],[1860,2990],[34,2960]] },
"box": { "xywh_px": [1452, 1880, 386, 560], "row": 2, "slot": 5, "detector_conf": 0.81 },
"what": { "product": "p-104223", "category": ["Grocery", "Pasta", "Short pasta"],
"pack": "pouch", "ocr": "MACARONI BIO" },
"valid": { "class": "valid_product", "p": 0.99 },
"decisions": [
{ "actor": "model", "pick": "p-209871", "visual_sim": 0.78, "top1_top2_gap": 0.03 },
{ "actor": "jev", "pick": "p-104223", "p": 0.72, "verdict": "review" },
{ "actor": "human", "role": "manager", "shown": ["p-104223", "p-209871", "p-118802"],
"action": "switch", "chosen": "p-104223", "latency_ms": 3900 }
],
"sales": { "product": "p-104223", "window": "30d", "units_per_day": 4.2 }
}
3. What we hold today
The reference store is a full-size grocery store in Southeast Asia with about 250 bays across 16 departments. Every bay has been photographed.
| Holding | Count |
|---|---|
| Product boxes on shelf photos | ~41,500 |
| Distinct products identified on shelf | 8,692 |
| Boxes with full verifier records (validity, product, decision) | 31,660 |
| Shelf crops with a pack-type label | 8,751 |
| Catalogue products, each with a reference image | 30,000+ |
| Category tree (L1 / L2 / L3) | 16 / 95 / 391 |
| Product-days of till sales | 1.4M |
| Days of sales history | 295 |
| Products with sales | ~26,000 |
Table 2. Current holdings, reference store.
3.1 What the boxes are
A detector's boxes are not all products. Our verifier classified every box in the reference store; the non-product classes are the hard negatives a shelf model has to learn.
| Validity class | Share of boxes |
|---|---|
| Valid product | 82.3% |
| Too small or blurry | 11.5% |
| Several products in one box | 2.9% |
| Partial or occluded | 2.0% |
| Price tag or label | 1.1% |
| Shelf edge or fixture | 0.1% |
| Empty shelf, other | <0.1% |
Table 3. Validity classes over 31,660 boxes.
3.2 What the products look like
| Pack type | Share of confirmed crops |
|---|---|
| Pouch or bag | 39.1% |
| Box | 21.5% |
| Bottle | 19.5% |
| Jar | 10.9% |
| Can | 8.8% |
Table 4. Pack types over 8,751 confirmed shelf crops. Pouches, the hardest to recognise because they deform, are the largest group.
Packaging is bilingual, the local language and English. Near-variants, the same product in another flavour or size, are the most common confusion and the reason every decision record keeps the alternatives that were shown.
4. How it is collected
- Associates, not annotators. Store staff photograph shelves with a phone app during normal walks. The app ties each photo to a bay on the floor plan and records the shelf outline the associate draws. No special hardware.
- Model first, people last. A detection and matching pipeline proposes a product for every box. A verifier, Jev, settles the clear cases and classifies non-products. People see only what is left, in a review screen built for speed, and every action is logged before and after.
- Nothing is overwritten. A new photo of a bay is a new version. Labels are appended, never edited. Original photos are never deleted.
- Sales arrive daily. Till exports are imported every morning and joined by product id.
The method is described in full in Shelf-1: learning products from the shop floor.
5. Growth estimate
Scaling the reference store's rates (one photo per bay per walk, about 102 boxes per photo, the verifier's current triage shares) to ten stores:
| Per walk | Per year, weekly walks | |
|---|---|---|
| Shelf photos | ~2,500 | ~130,000 |
| Product boxes | ~256,000 | ~13M |
| Boxes settled by the verifier at ~98% precision | ~68,000 | — |
| Labelled non-product and hard boxes | ~40,000 | — |
| Product-days of sales | — | ~17M |
Table 5. Projected growth. Assumes the reference store is typical of the other nine. After the first fully reviewed walk, most boxes carry their label forward; later walks add new versions of each bay over time.
6. What it is for
- Dense detection in grocery. Fine-tuning detectors on real Southeast Asian shelves, with explicit negatives for price tags, fixtures and merged boxes.
- Product recognition with hard negatives. Every human decision comes with the ranked alternatives that were rejected, which is what metric learning needs.
- Knowing when to trust a model. Verifier probabilities and human outcomes on the same boxes train decision models that route work between a model and a person.
- Shelf to sales. Whether a gap, a misplaced product or a low facing count moves revenue, measured on the same bay, product and day.
- Benchmarking. Held-out stores and weeks with human-only labels, versioned against the exact system that produced each guess.
7. Access
ShelfTrace is not released and will not be open-sourced. It trains and evaluates Shelf-1. Retail partners who contribute stores receive models trained on their own shelves and the enriched records for their stores. AI labs and vision teams can ask about evaluation access and licensing.
Contact founders@generalmachines.ai.
8. Limitations
- One chain, one country so far. Formats, fixtures and packaging elsewhere will differ.
- Mixed label sources. Labels come from people, the verifier and models. Every label says which; human labels are the only ones used for testing.
- One reference image per product. Catalogue images are studio shots; shelf crops of confirmed matches are added to the gallery over time.
- Revisits are starting. Most bays have one photo so far. Repeat walks of the same bays begin with weekly capture.
- People. Photos are of shelves. People are not labelled, and any person in frame is blurred before any partner access.
References
- E. Goldman, R. Herzig, A. Eisenschtat, J. Goldberger, T. Hassner. Precise Detection in Densely Packed Scenes. CVPR 2019. arXiv:1904.00853 (SKU-110K)