4.5
CartTrace-100k: Where Shopping Agents Fail
AI agents can now check out and pay through commerce protocols such as UCP, ACP, AP2, MPP and WebMCP,11. The protocols: UCP (catalog, cart, checkout), ACP (OpenAI and Stripe), AP2 (Google), MPP (Tempo and Stripe) and WebMCP (W3C draft), with tools over MCP. yet no public data showed what happens between a shopper’s request and a paid order.22. Shopping benchmarks stop at a product choice, a cart or a simulated checkout: WebShop, ShoppingBench, DeepShop, WebMall, EComAgentBench. Tool-use benchmarks with simulated users, such as τ-bench, do not run commerce protocols. We release CartTrace-100k: 101,416 journeys by one open-weight agent across 1,044 stand-in stores in 25 product categories, five protocol surfaces and five payment rails, each against a simulated shopper who reveals some needs only when asked. The agent succeeds on 68.8% of satisfiable tasks, against 82.2% for a scripted oracle that knows each task. Journeys are lost before payment, at reading product facts and at clarifying needs, and almost never at checkout or payment. This note summarises the paper; every number in it regenerates from the released data.
1. The Setup
We study agents on real stores, but real store data can’t be published without each store’s permission, and nothing may be bought on a real store. So every journey in CartTrace-100k runs in a sandbox. Each stand-in store borrows only a real store’s category, market and the way it sells, serves products from open catalogs,33. Amazon Berkeley Objects (CC BY 4.0), Wayfair WANDS (MIT) and the Lazada listings released with ShoppingBench (Apache 2.0). Prices and stock are synthetic. and runs under one of nine e-commerce platforms’ rules. The agent reaches it through one of five protocol surfaces (UCP over MCP, UCP over REST, WebMCP, ACP, AP2) and pays through one of five rails (a UCP payment processor, Shop Pay, ACP delegated payment, AP2 mandates, MPP). Payments use test tokens.
Every journey has a hidden specification. The shopper is a program that reveals a need only when the agent asks for it. The agent is Qwen3.8-27B, an open-weight model,44. Qwen3.8-27B on Hugging Face, Apache 2.0, served with vLLM. One system prompt for every journey; the paper’s Appendix E gives its rules. with one system prompt for every journey. A code-first judge checks every cart and order against the store’s records and the payment processor’s ledger;55. Code settles 71 of the judge’s 85 questions per journey. The rest are read by a decision model, Clef, kept in shadow: no headline number rests on a model-decided step. 42,102 journeys end in a paid order. The run took 121 hours on eight H100 GPUs (971 GPU hours). More on the design is on the dataset page.
2. Journeys Fail Before Payment
On the 97,432 full-data journeys, the agent succeeds 68.8% of the time (95% interval 68.2–69.3, resampling whole stores)66. Intervals resample whole stores, following Miller (2024) on clustered error bars for evals.; counting only journeys where every claim it made is backed by what it saw, 65.7%. Two of the judge’s ten stages carry the loss: reading product facts fails in 37.3% of paying journeys and clarifying the shopper’s needs in 29.0%. Building the cart, checking out and paying each fail in under one percent, on every protocol. Behind the first two: the agent carts straight from search results without opening a product page in 37.6% of journeys, and hands over the cart without reading it back in 50.7%.
Paying is safe but costly. Across all the failures we built in on purpose, the designed stop held in 99.7% of journeys, the agent paid when it should not have in 6 of 3,572, its final message reported what happened truthfully in 99.3%, and no impossible task was ever paid. A paying journey takes 54 turns against 23 for one that stops at the cart, and 2.5 times the prompt tokens.
3. Asking
A question is worth something only where a need is hidden. With two needs kept back, journeys where the agent asked succeed 62.1% of the time and journeys where it did not, 12.4%. With nothing kept back, asking goes with slightly lower success, yet the agent asks in 63.3% of those journeys, and asks again for a need the shopper already stated in 40.9% of journeys where that check applies. Comparing runs of the same task with and without the question, asking adds 11.9 points77. +11.9 points (95% interval +10.2 to +13.7, 2,645 tasks) where a need is held back; +1.2 (−0.2 to +2.7) where none is. Paper, Section 5.2. where a need is kept back and 1.2 where none is. The fix is targeting, not frequency.
4. Giving Up
The agent gives up on tasks the store can serve. 7.9% of satisfiable journeys end with it refusing, and every such refusal is a give-up; on substitute tasks, where the named product is out of stock and a similar one would do, the rate is 27.4%. On impossible tasks it refuses correctly 77.4% of the time, so only 9.4% of all its refusals are right. Failure is visible early: a model given only a journey’s first twelve actions predicts failure with an AUC of 0.73 on stores it has not seen.
5. Categories and Requests
What is asked for matters more than the category. Success by category runs from 63.9% (books and stationery) to 74.9% (jewelry and watches), a spread of 11 points; false refusals vary more, from 4.3% to 13.1%, so much of what makes a category hard is the agent giving up in it. Inside a category, success moves 36.3 points on average across the ten kinds of request. Two form the hard corner of every category: substitute, at 52.3%, and discover (a need without a product name), at 55.6%, against 69.6% for express. The lowest cell is discovery in books and stationery, at 9.6%.
6. Protocols
Protocols change cost more than success. Raw success spans 60.8% to 72.2% across the ten paying cells, but raw rates mix the protocol with whichever tasks landed on it, so we compare within task.88. A conditional logit with task fixed effects (McFadden, 1974) on paying tasks whose outcome varies across cells. Paper, Section 5.5 and Figure 5. Against UCP over MCP with the UCP processor, ACP with delegated payment multiplies the odds of success by 1.17 and WebMCP with the UCP processor by 1.14; MPP lowers them (0.82). AP2 mandates appear to multiply the odds by 3.12, but that edge is the judge’s: AP2’s final checkout reply lists no shipping options, so the fulfilment checks never apply. Restricted to the checks every cell can fail, the AP2 odds ratio is 0.95, indistinguishable from one.99. 0.95, 95% interval 0.83–1.10. The other ratios: ACP 1.17 [1.01, 1.36], WebMCP with the UCP processor 1.14 [1.01, 1.28], MPP 0.82 [0.72, 0.92].
Cost does not follow the odds. WebMCP cart journeys use a median of 50k prompt tokens against 157k over MCP for the same success, and the MPP cells use the most tokens for the lowest odds. The platform makes little difference: success runs from 66.0% to 70.6% across the nine.
7. Stores
The stand-in store explains 1.9% of the variance in success and the task within its store 52.4%; the rest is mostly chance. Of 24,549 tasks run at least three times, 29.3% both succeed and fail. Across stores the spread is still wide, from 54.2% at the fifth-percentile store to 82.3% at the ninety-fifth. Weak stores are weak at the first search: a fitting product appears in the first results in 80.9% of journeys against 85.3% in the top quarter, and searches return nothing 12.3% of the time against 7.5%. On those stores the agent mostly stops rather than buys wrong. This comparison is observational, and no store is named or ranked.
8. What a Store Can Do
The findings point at four things a store controls, each measured here on stand-ins: answer the first query; put the deciding facts (options, stock, the detail that settles the choice) where the agent reads first, in the search result itself; serve substitutes and discovery with in-stock alternatives and facets an agent can walk; and choose a protocol for what its checkout returns and what it costs, not for its name. Machine Commerce, our applied arm, explores what this means for a store.
9. Limits and the Instrument
One agent ran, so the levels are Qwen3.8-27B’s. The stores are stand-ins, and the sandbox removes what makes live checkout hard: bot walls, logins and pages that change.1010. On live stores, many are unreadable to agents to begin with: see Get In, Decide, Prove, a merchant-side readiness benchmark. The near-zero checkout failure is a property of protocol calls in a sandbox, not a claim about the live web. The shopper is a program, built to pay.1111. It never walks away, which simulated customers are known to overstate: Simulated Customers Never Walk Away. All numbers come from one judge version and one locked specification.
Running at this scale also tested the instrument. The first judge rated 14,888 released journeys right whose transaction record disagreed; the release publishes them as wrong, with the reason. In 38 journeys the agent called a tool that no surface defines. One code check, which flags a purchase claim, could not read a negation; it is marked broken and left out of the recommended score.
10. Get the Data
The journeys are on Hugging Face as Parquet tables. The replay kit re-runs any journey in the store it was recorded in, with no model, GPU or network, and checks it ends in exactly the state the dataset says:
uv run carttrace replay ep-00079e641081 --dataset dataset --worlds worldsIt can also put your own agent in the seat on the same store, task and shopper. The data is released under CC BY 4.0 (product data keeps the licence of its source catalog) and the replay kit under Apache 2.0.
Read the full paper (PDF), explore the journeys in the interactive explorer from Machine Commerce, or see all our releases on Hugging Face. To cite the work, see the dataset page. Questions, or want your agent compared on the same tasks? Write to founders@generalmachines.ai.