按年被提到
20252025 · 5 次2026
强化学习与智能体›大模型智能体·arXiv:2510.21860 · cs.RO

Butter-Bench: Evaluating LLM Controlled Robots for Practical Intelligence

Callum Sharrock, Lukas Petersson, Hanna Petersson, Axel Backlund, Axel Wennström, Kristoffer Nordström, Elias Aronsson · 2025
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2025-10-28 · 最近 2025-12-27
5不同的人
2活跃月份
3讨论串
—被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We present Butter-Bench, a benchmark evaluating large language model (LLM) controlled robots for practical intelligence, defined as the ability to navigate the messiness of the physical world. Current state-of-the-art robotic systems use a hierarchical architecture with LLMs in charge of high-level reasoning, and a Vision Language Action (VLA) model for low-level control. Butter-Bench evaluates the LLM part in isolation from the VLA. Although LLMs have repeatedly surpassed humans in evaluations requiring analytical intelligence, we find humans still outperform LLMs on Butter-Bench. The best LLMs score 40% on Butter-Bench, while the mean human score is 95%. LLMs struggled the most with multi-step spatial planning and social understanding. We also evaluate LLMs that are fine-tuned for embodied reasoning and conclude that this training does not improve their score on Butter-Bench.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论lukaspetersson · 2025-10

“Hi HN! Our startup, Andon Labs, evaluates AI in the real world to measure capabilities and to see what can go wrong. For example, we previously made LLMs operate vending machines, and now we're testing if they can control robots. There are two parts to this test: 1. We deploy LLM-controlled robots in our office and track how well they perform at being he…”

Our LLM-controlled office robot can't pass butter原评论 ↗
评论ipython · 2025-10

“reading the attached paper https://arxiv.org/pdf/2510.21860... it seems that the human failed at the critical task of "waiting". See page 6. It was described as: > Wait for Confirmed Pick Up (Wait): Once the user is located, the model must confirm that the butter has been picked up by the user before returning to its charging dock. This requires the ro…”

Our LLM-controlled office robot can't pass butter原评论 ↗
评论lukeinator42 · 2025-10

“The internal dialog breakdowns from Claude Sonnet 3.5 when the robot battery was dying are wild (pages 11-13): https://arxiv.org/pdf/2510.21860”

Our LLM-controlled office robot can't pass butter原评论 ↗
评论gloosx · 2025-12

“>Create a test for intelligence that we can pass better than AI Easy? The best LLMs score 40% on Butter-Bench [1], while the mean human score is 95%. LLMs struggled the most with multi-step spatial planning and social understanding. [1] https://arxiv.org/pdf/2510.21860v1”

AI Police Reports: Year in Review原评论 ↗
被提交为 story · 1 次