Shelf-1: learning products from the shop floor

How a ground-staff photo app, a probabilistic verifier and a human review environment turn routine shelf photos into training data for a retail vision model.

Status: In development. Shelf-1 is trained and evaluated on ShelfTrace, our private shelf dataset.

Abstract. Shelf analytics fails on the shop floor, not in the lab. A detector trained on public shelf images reaches 0.947 mAP50 on its own validation set. On real store photos, 14.8% of its boxes are invalid, and our matcher picks the right product only 55.6% of the time against a catalogue of more than 30,000 products. We describe the system we built to close that gap. Store associates photograph shelves with a phone app. A pipeline detects and matches every facing. A probabilistic verifier, Jev, triages each detection; at confidence 0.9 or higher it settles half the boxes with 98.9% precision. The rest go to a review environment built so that anyone can validate a match quickly. Every decision is logged with the full state, the candidates the person was shown and the action they took, so that it can be used for supervised, ranking and reinforcement-style training. We estimate each store walked yields about 25,000 labelled product observations. Shelf-1 is the model family trained from that loop: a detector, a matcher and a decision model.

1. The problem

A grocery shelf is one of the hardest scenes in commercial computer vision. Products are packed edge to edge. Hundreds of near-identical variants differ only by flavour, size or a line of local-language text. Lighting changes from aisle to chiller. The products themselves change every week.

Public datasets solve part of this. SKU-110K¹1. SKU-110K: Goldman et al., Precise Detection in Densely Packed Scenes, CVPR 2019. Code and data: SKU110K_CVPR19. taught detectors to find densely packed objects, but it labels every object with one class, "product". It says nothing about which product, and nothing about the stores we serve. The data that would close that gap, real in-store photos labelled at the product level, does not exist publicly and is expensive to make by hand.

Our bet is that the store already produces it. Associates walk the floor every day. If the walk produces a photo, and the system produces a guess, a quick human or machine judgement on that guess is a label. The work is in making that judgement cheap, trustworthy and recorded in a form a model can learn from.

2. System overview

The deployment we describe runs in a grocery chain in Southeast Asia. The reference store has about 250 bays across 16 departments and a catalogue of more than 30,000 products, each with one reference image from the retailer's product feed. Sales from the tills are joined to the shelf, so every facing can be ranked by revenue.

3. The ground-staff app

The capture app is built for associates, not annotators. It runs on any Android or iOS phone. The associate picks the store and the department, takes the photo, and drags four corners to mark the shelf. That polygon travels with the photo, so everything outside the shelf is ignored downstream and later masked out of training labels.

Three properties matter for data quality:

• Every photo has an address. A photo belongs to a store, a department and, on the floor map, a bay. The same bay photographed next week is a new version of the same place, never an overwrite. That gives us the same shelf over time for free.
• Nothing is lost. Uploads queue on the phone and retry with backoff until the server confirms them. The server deduplicates by shelf and content hash, so a retry never creates a second copy. Original photos are never deleted from storage.
• The capture itself is a label. In the map-first version of the app, now rolling out, the associate taps where they stand and picks the bay from the nearest few. We record what was suggested, what was picked and how long it took. These are weak labels for placing photos on the plan.

4. Detection and matching

4.1 The detector

The detector is YOLO26-L, initialised from COCO-pretrained weights and fine-tuned on SKU-110K as a single "product" class: 50 epochs at 640 px on one GPU, about 16 hours. On the SKU-110K validation split it reaches:

Table 1. Detector after fine-tuning on SKU-110K.

In our stores the picture is different. On a frozen set of store photos, 14.8% of raw boxes are invalid: tiny boxes, boxes nested inside other boxes, duplicates, and products seen through the shelf in the next aisle. Four cheap geometric filters (minimum area, a background-scale cut at 0.45 of the median box height, nested-box removal, duplicate suppression) bring that to 5.9% while losing 0.2 points of recall. A stricter setting reaches 0.35% at a cost of 1.8 points. What the filters cannot fix are price tags and shelf labels that look like packs. That needs labels from the store, which is what the rest of this system produces.

4.2 Matching a box to a product

Each box is padded by 8%, embedded with SigLIP ViT-L/16 at 384 px²2. SigLIP: Zhai et al., Sigmoid Loss for Language Image Pre-Training, ICCV 2023., and searched against the catalogue in an exact inner-product FAISS index³3. FAISS: Johnson, Douze and Jégou, Billion-scale similarity search with GPUs, IEEE Transactions on Big Data, 2019., returning the top 25. A small penalty pushes down "hub" products that match everything; in one audit four products took 11.5% of all matches. A department guard drops candidates whose category cannot be on that shelf. OCR adds a bonus when the product's brand appears on the pack. A match is withheld ("abstain") when similarity is below 0.70 without a brand match, or below 0.80 for an off-category candidate.

OCR is harder than it looks. A general-purpose cloud OCR could not read the local script at all on our crops. An open model trained for that script raised brand recall only modestly, from 23.5% to 25.5% when the two were combined. Text is evidence, not an answer.

Withholding is deliberate. In our store-wide data, roughly 69% of boxes receive a product and 27% are withheld for low similarity. A confident wrong answer is worse than an honest "not sure", because only the second one reaches a person.

5. Jev: cutting bad finds before a person sees them

Most detections are fine, and a person's time is the scarcest resource in the loop. Jev is a probabilistic decision model: given a structured state and a set of typed questions, it returns a probability for each answer rather than free text. We use it as a verifier on every detection.

5.1 What Jev sees

• The shelf crop, and the catalogue photo of each candidate product (the pipeline's top five plus nearest visual neighbours, seven options with "none of these").
• Evidence: department, bay and shelf row, the box's height against the row median, aspect ratio, detector confidence, the text read from the pack, and the products picked either side.
• Per candidate: visual similarity and rank, name similarity, OCR overlap with the name or brand, whether the product belongs in that department, and whether it sold in that store in the last 90 days.

5.2 What Jev answers

For each box Jev answers three questions: is this a valid product, and if not, what is it (price tag, empty shelf, fixture, cut off, several products, too small or blurry); which candidate is it; and does the pack type (can, bottle, box, pouch, jar) agree with the catalogue photo. We ask the product question in two independent views, as a set of yes/no checks per candidate and as a shuffled multiple-choice sheet, and act only when both agree.

Table 2. Triage policy (v2).

5.3 How well it works

On a graded evaluation set of 1,062 detections:

Table 3. Jev on the graded set. Where Jev disagreed with a human label, its confidence was always below 0.7. On "hub" boxes, where the pipeline falls back to a product that matches everything, the pipeline was right 11.3% of the time and Jev 56.6%.

Run over a whole store, 309 photos and 31,660 boxes, Jev cost about five US dollars at list price and about 1.8 seconds per call, run in parallel. It caught price tags and envelope labels the pipeline had matched to products, near-variant swaps such as one pasta shape for another, and a dog treat matched where the pack was the cat version. Of the boxes, it confirmed or corrected 27%, removed 2%, set aside 14% as unusable, and sent 58% to people. As a by-product it split the detector's output into 8,751 confirmed products, 4,052 false positives and 18,851 uncertain boxes: a ready-made detector feedback set from one store.

The first version of the policy removed every unusable box. Reviewing the result, we changed it: a box that is cut off or blurry is still a real product on a real shelf. In v2 those boxes stay on the planogram and feed only the detector. Removals fell from 4,890 to 590.

Jev is used only to make labels. It is never in the inference path, so a deployed Shelf-1 model needs no external call.

6. The review environment

Whatever Jev does not settle goes to people. We built the review screen as an environment in the reinforcement-learning sense, while keeping it simple enough for a store manager on a phone.

6.1 State, action, outcome

• State. The product crop, with its outline drawn inside a wider cut of the shelf; the full photo; the text read from the pack; and up to six ranked candidates with their catalogue photos and scores. Nothing is pre-selected.
• Actions. Pick a candidate, then confirm it as a separate step; mark "not a product"; or skip. Each choice has a keyboard shortcut and a large button on a phone. A decision waits five seconds with an undo before it is written, so a slip never becomes a label.
• Outcome. Once the undo window passes, the decision updates the store's live planogram, so the reviewer sees their work reflected in the store's revenue map.

The queue is ordered by expected value of a human look: boxes with no product but real candidates first, then Jev's unsure cases and category clashes, then low similarity. Within a kind, the least certain go first.

6.2 What one decision records

Every action, human or machine, is appended to one table that is never edited. The state is captured before the change, so the record holds the model's opinion the person reacted to, not their own answer.

context    image (storage key, hash, size, store, bay, version, crop polygon)
           box (pixels and normalised, detector confidence, row, slot)
           prediction (model's pick, similarity, fusion score, top-1/top-2 gap,
                       OCR text, abstain reason, verifier verdict)
           pipeline (run, model versions, code version, index hash)
shown      ranked candidates with scores, as presented
chosen     the action and its label: confirm | switch | not_product | edit_box | add_box | …
actor      human | jev | llm | ml, with role and model version
timing     client time, latency

Because the record holds the candidates as shown, with their scores, the log supports more than supervised learning. Each decision is a contextual-bandit sample, a context, the ranked arms offered, the arm chosen. A confirm or switch is a positive pair; every other shown candidate is a ranked hard negative; "not a product", a deleted box and a drawn box are detector negatives and missed positives. Human, Jev and LLM decisions sit in the same table, marked by actor, so we can weight them differently and measure each against the others.

6.3 The decision model

The environment has one more participant: the policy that decides who should judge a box. Today it is a fixed rule (Table 2). The trail lets us learn it: for each kind of box, how often does Jev's confident answer survive a human check, and how much human time does it save? The decision model's target is the lowest cost of people's time at a fixed label precision.

7. How much data this produces estimate

The measured inputs are from one store: about one photo per bay per walk, about 102 boxes per photo after filtering, and the Jev triage shares in section 5.3. We scale those to ten stores. Every figure in this section is a projection and depends on the stated assumptions.

Table 4. One full walk, projected. Assumes the reference store's bay count and box density hold in every store.

Three ways to finish the review bucket:

Table 5. Assumes about 4 seconds per human decision with keyboard shortcuts. A vision-language teacher we tested agreed with human labels 84% of the time; the hybrid share sent to people is an assumption to be measured.

The first walk is the expensive one. Shelves change slowly: most products sit in the same slot next week. A verified label carries forward to the same product in the same slot, so later walks mostly re-confirm and only changes need a look. If 10–20% of boxes change week to week, the human load in steady state is roughly two to four hours a week per store.

Over a year of weekly walks, ten stores produce on the order of 130,000 shelf photos and 13 million product observations, each tied to a bay, a date and the till sales of that product in that store. The first fully verified walk of ten stores alone gives about 256,000 labelled boxes and, from the shown candidates, a much larger set of ranked hard-negative pairs.

8. Shelf-1

Shelf-1 is the model family trained from this trail. It has three parts, each with a clear training signal:

Table 6. Shelf-1 components.

Distilling Jev into Shelf-1 Decide is what lets the deployed model run on the retailer's own hardware with no external calls, while Jev keeps labelling in the loop.

8.1 Evaluation protocol

• Held-out stores and held-out weeks. No store or week appears in both training and test.
• Human labels only for test. Jev and LLM labels are training signal, never ground truth.
• Versioned baselines. Every label records the detector, embedding model, index and code version that produced the guess, so results compare against the exact system that was running.
• Metrics. Detection mAP50 and mAP50-95 on store photos; product top-1 accuracy; coverage at fixed precision (95%, 99%) for Shelf-1 Decide; human seconds per correct label.
• Business check. Whether shelf findings predict sales, using the till data joined to each bay.

9. Limitations

• Graded labels. Part of the graded set in section 5.3 was labelled by a vision-language teacher that agrees with people 84% of the time. Precision figures should be read with that in mind until they are repeated on a fully human-labelled set.
• One reference store. The projections in section 7 assume the reference store is typical. Store size, layout and product mix will move them.
• Human speed is assumed. We have not yet measured seconds per decision at scale or agreement between reviewers. Both are recorded by the environment and are the first things we will report.
• Catalogue images. Each product has one reference image, often a studio shot that differs from the shelf view. The matcher is limited by that until shelf crops from confirmed matches join the gallery.
• Detector not yet adapted. The detector has not yet been fine-tuned on store photos. Section 8 describes the method; results will follow.

10. Conclusion

A store produces training data every day; it just doesn't keep it. A phone app that gives every photo an address, a verifier that settles the easy half, and a review screen that records exactly what a person saw and did turn routine shelf walks into the labelled, in-store data that public datasets lack. Shelf-1 is what we train on it.

References

1. E. Goldman, R. Herzig, A. Eisenschtat, J. Goldberger, T. Hassner. Precise Detection in Densely Packed Scenes. CVPR 2019. arXiv:1904.00853 (SKU-110K)
2. X. Zhai, B. Mustafa, A. Kolesnikov, L. Beyer. Sigmoid Loss for Language Image Pre-Training. ICCV 2023. arXiv:2303.15343 (SigLIP)
3. J. Johnson, M. Douze, H. Jégou. Billion-scale similarity search with GPUs. IEEE Transactions on Big Data, 2019. arXiv:1702.08734 (FAISS)
4. N. Ravi et al. SAM 2: Segment Anything in Images and Videos. 2024.
5. Ultralytics YOLO26. Ultralytics, 2025.
6. PaddleOCR. PaddlePaddle.