Qwen released E-Commerce Bench, a test where AI agents run a simulated online shop for a year. Each agent starts with ¥100,000 and opens up to four stores. It must negotiate with suppliers, set prices, hold stock and stay solvent. Nothing resets, and the agent's memory overflows long before day 365. Qwen ran 18 models five times each and scored them on end-of-year assets. Rankings differ sharply by dimension. The top earner sent 18.5% of its buying cash to fraudulent suppliers, against 0.12% for the most careful model. Several models went bankrupt in some runs. Demand and supplier pricing are fixed in advance, so results reflect the agent, not luck. Of 576 suppliers, 152 run one of five scam patterns.
What changed
Agent benchmarks mostly tested short tasks that reset after each attempt.
What it unlocks
Running an open-source year-long business simulation to compare how models handle money, suppliers and memory.
- ¥100,000 starting balance, 365 days
- 18 models, 5 episodes each
- best model ended at ¥1,431k mean
- 18.5% vs 0.12% spend to scam suppliers
What you need to act on it
- Python 3.10+
- API key for the model under test
- a second key for the supplier role-play model
Sources