按年被提到
20192023 · 3 次2026
大模型与语言模型›大模型评测与真实性·arXiv:1905.07830 · cs.CL

HellaSwag: Can a Machine Really Finish Your Sentence?

Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, Yejin Choi · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2023-03-05 · 最近 2023-12-10
3不同的人
2活跃月份
3讨论串
407被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Recent work by Zellers et al. (2018) introduced a new task of commonsense natural language inference: given an event description such as "A woman sits at a piano," a machine must select the most likely followup: "She sets her fingers on the keys." With the introduction of BERT, near human-level performance was reached. Does this mean that machines can perform human level commonsense inference? In this paper, we show that commonsense inference still proves difficult for even state-of-the-art models, by presenting HellaSwag, a new challenge dataset. Though its questions are trivial for humans (>95% accuracy), state-of-the-art models struggle (<48%). We achieve this via Adversarial Filtering (AF), a data collection paradigm wherein a series of discriminators iteratively select an adversarial set of machine-generated wrong answers. AF proves to be surprisingly robust. The key insight is to scale up the length and complexity of the dataset examples towards a critical 'Goldilocks' zone wherein generated text is ridiculous to humans, yet often misclassified by state-of-the-art models. Our construction of HellaSwag, and its resulting difficulty, sheds light on the inner workings of deep pretrained models. More broadly, it suggests a new path forward for NLP research, in which benchmarks co-evolve with the evolving state-of-the-art in an adversarial way, so as to present ever-harder challenges.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论rnosov · 2023-03

“You can read the original LLaMA paper which is pretty accessible[1]. For example, they claim to outperform GPT-3 on HellaSwag benchmark ( finishing sentences ). You can find examples of unfinished sentences in the HellaSwag paper [2] on page 13. Unfortunately for LLaMA, most people would be probably just asking questions about Captain Picard and so on, an…”

Show HN: Llama-dl – high-speed download of LLaMA, Facebook's 65B GPT model原评论 ↗
评论namuol · 2023-03

“FWIW, the “HellaSwag” benchmark from the GPT-4 paper consists of many such “riddles” and it achieves ~95% accuracy (essentially par with humans). The next best model (LLM or otherwise) was around 85%. It is pretty remarkable. GPT-4 paper: https://arxiv.org/pdf/2303.08774.pdf Original HellaSwag paper from 2019: https://arxiv.org/pdf/1905.07830.pdf”

On the left is GPT-3.5. On the right is GPT-4原评论 ↗
评论recursivecaveat · 2023-12

“They used to be quite bad at common-sense-reasoning type tasks like HellaSwag: https://arxiv.org/pdf/1905.07830.pdf but it seems like they're creeping up into the high-80s for % of responses correct now.”

Three Things That LLMs Have Made Us Rethink原评论 ↗