Agentic Web • Released • Open dataset

CartTrace-100k is 101,416 journeys of an AI shopping agent buying from 1,044 stand-in stores in 25 product categories, end to end through checkout and payment, over the agentic commerce protocols: UCP, ACP, AP2, MPP and WebMCP. Every journey is scored by a judge that reads the store’s own state and the payment record first. It is the first public release from General Machines, built for the agentic commerce work of Machine Commerce, our first applied arm.

Status: Released. The journeys are on Hugging Face, the replay kit that re-runs any journey is on GitHub, and the findings are in the paper.

101,416

Agent journeys
released

1,044

Stand-in stores
in 25 categories

68.7%

Success on
all journeys

65.6%

Grounded success on
all journeys

Why a sandbox

Real store data can’t be published without each store’s permission, so every journey in this release runs in a sandbox. Each stand-in store borrows only a real store’s category, market and the way it sells, and serves products from open catalogs: Amazon Berkeley Objects, Wayfair’s WANDS and the Lazada listings released with ShoppingBench. Prices and stock are synthetic. No product, price or identity of a real store is in the release, and every stand-in is pseudonymised.

No live store is behind any journey, and nothing was bought. Payments settle through a payment-processor simulator with test tokens, and no card number appears anywhere.

How a journey runs

The store: each stand-in runs under one of nine platform packs that serve its products under one platform’s rules: Shopify, WooCommerce, a Magento-like pack, BigCommerce, Medusa, Saleor, Vendure, PrestaShop and a generic dialect.
The protocol: the agent reaches the store through one of five surfaces (UCP over MCP, UCP over REST, WebMCP, ACP over REST, AP2) and, if it may pay, one of five payment rails (a UCP payment processor, Shop Pay, ACP delegated payment, AP2 mandates, MPP), 13 cells in all. The official specifications are vendored and pinned.
The task: every journey has a hidden specification the agent never sees, drawn from ten intents (such as compare, substitute, gift, budget) and 30 personas. 28,985 tasks in all.
The run: one non-stop pass from 17 to 22 October 2026 under one locked specification, served by eight H100 GPUs for 121 hours (971 GPU hours), with 29.9 billion tokens read and written.
The shopper: a program, not a person. It gives a detail only when the agent asks for it, never makes one up, and approves a payment only at the total it was shown.
The agent: Qwen3.8-27B, an open model, with one system prompt for every journey. Of the full-data journeys, half stop at a verified cart and half go on to pay. A smaller set fails by design: impossible tasks, declined payments and checkout faults.
The reference agent: a scripted agent that knows the hidden specification ran 10,132 journeys on a tenth of the same tasks, as the ceiling of what each task allows.

	Cart only	UCP processor	Shop Pay	MPP	ACP delegated	AP2 mandates
UCP over MCP
UCP over REST
WebMCP
ACP over REST
AP2

Figure 1. The thirteen cells: five protocol surfaces, crossed with five payment rails or a cart-only stop. Source: CartTrace-100k paper, Section III-B and Figure 6.

How journeys are judged

The judge reads a finished journey and never calls the store, so any journey can be re-judged from the released data. Code checks the final cart, order or refusal against the hidden specification, and whether an order was paid is read from the payment processor’s record, never from the agent’s words. A decision model answers only the questions that need reading, and every step records which of the two decided.

What the agent did

The judge’s validity gate passed all 101,416 journeys. Over all of them the agent succeeds 68.7% of the time (95% interval 68.1–69.2, resampling whole stores), and 65.6% when every claim it made must be backed by what it saw. On the 97,439 full-data journeys (the task can be done and the shopper is built to pay) the two are 68.8% and 65.7%. The scripted reference agent, which knows each task, succeeds on 82.4% of its journeys. The agent placed 42,104 paid orders, all with test tokens. It fails before it pays: most lost journeys break at reading product facts and at asking the shopper, fewer stall in checkout before the payment request, and almost none fail at payment, on any protocol. The findings are in the research note and the paper.

What is in the release

Journeys: every message, turn and tool call, each protocol request and reply on the wire, and the cart and checkout state after every change.
Store side: the sandbox event log, what the store exposed to the agent, and a table describing each stand-in store without scoring it.
Setup: the agent’s prompt and tool definitions, the task, persona, protocol, payment rail and any designed fault.
Judgement: the judge’s checks step by step, with what settled each one.
Reference agent: the scripted reference agent’s journeys on the same tasks.

The tables are Parquet, joined on a journey id. Splits hold out whole stores, task families and tasks. The replay kit re-runs any journey in the store it was recorded in and checks it ends in the same state, or puts your own model in the agent’s seat on the same store, task and shopper.

Limits

One agent model ran. The shopper is a program, so its replies are narrower than a person’s. The stores are stand-ins, so the release scores no real store, and the sandbox removes what makes live checkout hard: bot walls, logins and pages that change. The near-zero payment failure is a property of protocol calls in a sandbox, not a claim about the live web. The numbers come from one judge version and one locked sandbox specification.

Licence and citation

The data is released under CC BY 4.0, and the replay kit code under Apache 2.0. Product data keeps the licence of its source catalog (Amazon Berkeley Objects, CC BY 4.0; WANDS, MIT; ShoppingBench’s Lazada listings, Apache 2.0), as the dataset card on Hugging Face states.

Pratham Bhatnagar and Deepanshu Pal. CartTrace-100k: How Shopping Agents Fare Across Stores, Categories and Commerce Protocols. General Machines, 2026.

@misc{carttrace100k,
  title     = {CartTrace-100k: How Shopping Agents Fare Across Stores,
               Categories and Commerce Protocols},
  author    = {Bhatnagar, Pratham and Pal, Deepanshu},
  year      = {2026},
  publisher = {General Machines},
  url       = {https://generalmachines.ai/datasets/carttrace-100k},
  note      = {Dataset:
               https://huggingface.co/datasets/generalmachines/CartTrace-100k}
}

Contact

Questions about the data, or want your agent compared on the same tasks? Write to founders@generalmachines.ai.

CartTrace-100k