按年被提到
20242024 · 4 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2409.01374 · cs.AI

H-ARC: A Robust Estimate of Human Performance on the Abstraction and Reasoning Corpus Benchmark

Solim LeGris, Wai Keen Vong, Brenden M. Lake, Todd M. Gureckis · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2024-11-14 · 最近 2024-12-20
3不同的人
2活跃月份
2讨论串
12被引(OpenAlex)
arXiv 摘要(候选层不写导读)

The Abstraction and Reasoning Corpus (ARC) is a visual program synthesis benchmark designed to test challenging out-of-distribution generalization in humans and machines. Since 2019, limited progress has been observed on the challenge using existing artificial intelligence methods. Comparing human and machine performance is important for the validity of the benchmark. While previous work explored how well humans can solve tasks from the ARC benchmark, they either did so using only a subset of tasks from the original dataset, or from variants of ARC, and therefore only provided a tentative estimate of human performance. In this work, we obtain a more robust estimate of human performance by evaluating 1729 humans on the full set of 400 training and 400 evaluation tasks from the original ARC problem set. We estimate that average human performance lies between 73.3% and 77.2% correct with a reported empirical average of 76.2% on the training set, and between 55.9% and 68.9% correct with a reported empirical average of 64.2% on the public evaluation set. However, we also find that 790 out of the 800 tasks were solvable by at least one person in three attempts, suggesting that the vast majority of the publicly available ARC tasks are in principle solvable by typical crowd-workers recruited over the internet. Notably, while these numbers are slightly lower than earlier estimates, human performance still greatly exceeds current state-of-the-art approaches for solving ARC. To facilitate research on ARC, we publicly release our dataset, called H-ARC (human-ARC), which includes all of the submissions and action traces from human participants.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论trott · 2024-11

“> But if someone has already hit the human-level threshold There is some controversy over what the human-level threshold is. A recent and very extensive study measured just 60.2% using Amazon Mechanical Turkers, for the same setup [1]. But the Turkers had no prior experience with the dataset, and were only given 5 tasks each. Regardless, I believe ARC-…”

Francois Chollet is leaving Google原评论 ↗
评论obblekk · 2024-12

“Human performance is 85% [1]. o3 high gets 87.5%. This means we have an algorithm to get to human level performance on this task. If you think this task is an eval of general reasoning ability, we have an algorithm for that now. There's a lot of work ahead to generalize o3 performance to all domains. I think this explains why many researchers feel AGI …”

OpenAI O3 breakthrough high score on ARC-AGI-PUB原评论 ↗
评论hypoxia · 2024-12

“It actually beats the human average by a wide margin: - 64.2% for humans vs. 82.8%+ for o3.... Private Eval: - 85%: threshold for winning the prize [1] Semi-Private Eval: - 87.5%: o3 (unlimited compute) [2] - 75.7%: o3 (limited compute) [2] Public Eval: - 91.5%: o3 (unlimited compute) [2] - 82.8%: o3 (limited compute) [2] - 64.2%: human averag…”

OpenAI O3 breakthrough high score on ARC-AGI-PUB原评论 ↗
评论hypoxia · 2024-12

“Many are incorrectly citing 85% as human-level performance. 85% is just the (semi-arbitrary) threshold for the winning the prize. o3 actually beats the human average by a wide margin: 64.2% for humans vs. 82.8%+ for o3.... Here's the full breakdown by dataset, since none of the articles make it clear -- Private Eval: - 85%: threshold for winning th…”

OpenAI O3 breakthrough high score on ARC-AGI-PUB原评论 ↗