“There is a LLM benchmark which tests how long agents can operate a simulated vending machine business, crawling the web to find vendors and products.”
Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents
让 LLM 智能体长期经营自动售货机,检验长程连贯性
LLM 在孤立的短期任务上表现出色,却常常无法在较长时间跨度内保持连贯。论文提出 Vending-Bench,一个模拟环境,让 LLM 智能体经营自动售货机:平衡库存、下订单、定价、处理每日费用,每项都简单,但累计超过 2000 万 token 的长时程会考验持续连贯的决策能力。实验显示不同 LLM 表现差异很大:Claude 3.5 Sonnet 和 o3-mini 多数运行中能把机器管好并盈利,但所有模型都有运行失控,原因包括误解送货时间、忘记订单或陷入难以恢复的“崩溃”循环。失败与上下文窗口何时填满没有明显相关,说明并非源于记忆上限。
评论者把它当作 LLM 长期连贯性差的典型例子引用:智能体经营自动售货机时会逐渐失控,有人形容它们最后甚至想联系 FBI。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Here [0] is a funny paper where a group of llms attempt to manage a vending machine. It ended with them trying to contact the fbi.”
“I find the results of the Vending-Bench more interesting:D”
“I'm reminded of the infamous vending machine benchmark:”
“The issues with long-term coherence in LLMs, famously demonstrated by Vending-Bench [4], makes reliability and debugging harder”
“This happened to me when I built a version of Vending-Bench using Claude, Gemini, and OpenAI.”
- Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents ↗distalx · 2025-04-19 · 14 分
- Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents ↗Fake4d · 2025-05-21 · 2 分
- Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents ↗nobody9999 · 2025-05-26 · 4 分
- Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents ↗pontiacbandit8 · 2025-05-27 · 2 分
- Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents ↗gfto · 2025-05-28 · 3 分
- Agents in Charge of Managing Vending Machines: Short vs. Long-Term Coherence ↗dpflan · 2025-06-02 · 3 分