“My latest one was about about engineering and LLMs about which I'm particularly excited: Mathematical Capabilities of ChatGPT.”
Mathematical Capabilities of ChatGPT
用研究生级数学数据集 GHOSTS 评测 ChatGPT 与 GPT-4
论文用公开数据集和自建数据集,测试两个版本的 ChatGPT(2023 年 1 月 9 日与 30 日)及 GPT-4 的数学能力,并发布 GHOSTS 与 miniGHOSTS 两个由在职数学研究者整理、覆盖研究生级数学的自然语言数据集。结果显示 ChatGPT 最适合充当查询事实的数学搜索引擎与知识库接口,GPT-4 还能处理本科水平,但在研究生难度上失败。作者认为总体数学表现远低于研究生水平,与媒体上关于其解题能力的正面报道(可能存在选择偏差)不符。
多条提及来自作者本人介绍自己的这项工作,其他评论者则把它当作 ChatGPT 数学推理能力偏弱的基准证据。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“My latest one was about about engineering and LLMs about which I'm particularly excited: Mathematical Capabilities of ChatGPT.”
“In February I published a paper on mathematics + ChatGPT and my colleagues, who were also on LinkedIn, told me it garnered quite a bit of attention.”
“the author of [3] where we introduced one of the first natural-language datasets that test graduate mathematics for LLMs”
“Otherwise, just use grad-level maths to confuse it into giving refunds I suppose [1].”
“It is known that LLMs perform incredibly worse in mathematical reasoning and understanding”
- Mathematical Capabilities of ChatGPT ↗ColinWright · 2023-02-04 · 1 分