按年被提到
20232023 · 2 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2304.03439 · cs.CL

Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4

Hanmeng Liu, Ruoxi Ning, Zhiyang Teng, Jian Liu, Qiji Zhou, Yue Zhang · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-04-10 · 最近 2023-10-17
2不同的人
2活跃月份
2讨论串
201被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Harnessing logical reasoning ability is a comprehensive natural language understanding endeavor. With the release of Generative Pretrained Transformer 4 (GPT-4), highlighted as "advanced" at reasoning tasks, we are eager to learn the GPT-4 performance on various logical reasoning tasks. This report analyses multiple logical reasoning datasets, with popular benchmarks like LogiQA and ReClor, and newly-released datasets like AR-LSAT. We test the multi-choice reading comprehension and natural language inference tasks with benchmarks requiring logical reasoning. We further construct a logical reasoning out-of-distribution dataset to investigate the robustness of ChatGPT and GPT-4. We also make a performance comparison between ChatGPT and GPT-4. Experiment results show that ChatGPT performs significantly better than the RoBERTa fine-tuning method on most logical reasoning benchmarks. With early access to the GPT-4 API we are able to conduct intense experiments on the GPT-4 model. The results show GPT-4 yields even higher performance on most logical reasoning datasets. Among benchmarks, ChatGPT and GPT-4 do relatively well on well-known datasets like LogiQA and ReClor. However, the performance drops significantly when handling newly released and out-of-distribution datasets. Logical reasoning remains challenging for ChatGPT and GPT-4, especially on out-of-distribution and natural language inference datasets. We release the prompt-style logical reasoning datasets as a benchmark suite and name it LogiEval.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论ben_w · 2023-10

“The numbers in Table 1 of this paper are worse than I remembered, but I would also note that humans would fail the "can they reason?" standard I quoted, given the grades the species gets on several of the tests: https://arxiv.org/pdf/2304.03439.pdf > even if you define "logic" as "logical puzzles encoded in natural language" and not "the thing that allo…”

Finland's ChatGPT equivalent begins to think in Estonian as well原评论 ↗
被提交为 story · 1 次