“Late reply but, LLMs are not bad at abstraction in general, just spatially focused abstraction. https://arxiv.org/abs/2212.09196 Also, how the data is presented matters a lot. Currently, LLMs handle the benchmark much better presented linearly in 1 dimension https://arxiv.org/abs/2305.18354”
LLMs and the Abstraction and Reasoning Corpus: Successes, Failures, and the Importance of Object-based Representations
Can a Large Language Model (LLM) solve simple abstract reasoning problems? We explore this broad question through a systematic analysis of GPT on the Abstraction and Reasoning Corpus (ARC), a representative benchmark of abstract reasoning ability from limited examples in which solutions require some "core knowledge" of concepts such as objects, goal states, counting, and basic geometry. GPT-4 solves only 13/50 of the most straightforward ARC tasks when using textual encodings for their two-dimensional input-output grids. Our failure analysis reveals that GPT-4's capacity to identify objects and reason about them is significantly influenced by the sequential nature of the text that represents an object within a text encoding of a task. To test this hypothesis, we design a new benchmark, the 1D-ARC, which consists of one-dimensional (array-like) tasks that are more conducive to GPT-based reasoning, and where it indeed performs better than on the (2D) ARC. To alleviate this issue, we propose an object-based representation that is obtained through an external tool, resulting in nearly doubling the performance on solved ARC tasks and near-perfect scores on the easier 1D-ARC. Although the state-of-the-art GPT-4 is unable to "reason" perfectly within non-language domains such as the 1D-ARC or a simple ARC subset, our study reveals that the use of object-based representations can significantly improve its reasoning ability. Visualizations, GPT logs, and data are available at https://khalil-research.github.io/LLM4ARC.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“For now, LLMs are a lot better when you express this kind of task in text, preferably as a 1D arrangement. GPT-4's performance on the Abstraction and Reasoning Corpus. https://arxiv.org/abs/2305.18354 Also, this model trained to predict image tokens seemed to be able to perform Raven's Matrices type questions when expressed as an inpainting task in conte…”
“Oh i've read them. The claim doesn't match up to reality. It's as simple as that. You can claim anything you want to. https://arxiv.org/abs/2305.18354 All these papers you keep linking do is at best point out the shortcomings of current state of the art LLMs. They do not in any way disprove their ability to reason. I don't know when the word reason star…”
- LLMs and the Abstraction and Reasoning Corpus ↗og_kalu · 2023-11-19 · 1 分
- LLMs and the Abstraction and Reasoning Corpus ↗famouswaffles · 2024-03-10 · 3 分