按年被提到
20222022 · 1 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2210.09261 · cs.CL

Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them

Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V. Le, Ed H. Chi, Denny Zhou, Jason Wei · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2022-10-27 · 最近 2023-09-08
2不同的人
2活跃月份
2讨论串
329被引(OpenAlex)
arXiv 摘要(候选层不写导读)

BIG-Bench (Srivastava et al., 2022) is a diverse evaluation suite that focuses on tasks believed to be beyond the capabilities of current language models. Language models have already made good progress on this benchmark, with the best model in the BIG-Bench paper outperforming average reported human-rater results on 65% of the BIG-Bench tasks via few-shot prompting. But on what tasks do language models fall short of average human-rater performance, and are those tasks actually unsolvable by current language models? In this work, we focus on a suite of 23 challenging BIG-Bench tasks which we call BIG-Bench Hard (BBH). These are the task for which prior language model evaluations did not outperform the average human-rater. We find that applying chain-of-thought (CoT) prompting to BBH tasks enables PaLM to surpass the average human-rater performance on 10 of the 23 tasks, and Codex (code-davinci-002) to surpass the average human-rater performance on 17 of the 23 tasks. Since many tasks in BBH require multi-step reasoning, few-shot prompting without CoT, as done in the BIG-Bench evaluations (Srivastava et al., 2022), substantially underestimates the best performance and capabilities of language models, which is better captured via CoT prompting. As further analysis, we explore the interaction between CoT and model scale on BBH, finding that CoT enables emergent task performance on several BBH tasks with otherwise flat scaling curves.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论riku_iki · 2023-09

“> +50% on Big Bench Hard its +50% on few tasks. It is also +50% not against sota, but some their weak baseline. For example they received large boost on object_counting task where their final result is 86% acc, while current sota from https://arxiv.org/pdf/2210.09261.pdf is 93%”

Large Language Models as Optimizers. +50% on Big Bench Hard原评论 ↗
被提交为 story · 1 次