引用mrbungie · 2024-09
“o1 did much better in PlanBench that models from other providers indicating better perf in that area [1][2]”
引它说明 o1 在 PlanBench 上明显优于其他厂商模型
用 PlanBench 评测 o1:规划能力大幅提升,但远未饱和
论文用作者在 2022 年提出的可扩展规划基准 PlanBench,评估当前 LLM 以及 OpenAI o1 这类“大型推理模型”(LRM)。结果显示 o1 比竞争对手有质的提升,但仍远未让基准饱和;作者还指出这种提升引出了准确性、效率和可保证性等部署前必须考虑的问题。
评论者用它讨论 LLM 能否做规划:o1 明显优于其他模型,但名称随机化后多数模型表现很差,被当作模型靠记忆而非推理的证据。
“o1 did much better in PlanBench that models from other providers indicating better perf in that area [1][2]”
“For example, papers like this call into question whether or not a LLM can plan:”
“The vast majority of LLMs score pathetically low in this, but much better in non-randomised versions, very clearly demonstrating excellent memorisation skills, but pitiful reasoning ability.”