按年被提到
20172017 · 2 次2026
大模型与语言模型›大模型评测与真实性·arXiv:1707.07328 · cs.CL

Adversarial Examples for Evaluating Reading Comprehension Systems

Robin Jia, Percy Liang · 2017
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2017-09-16 · 最近 2018-01-15
3不同的人
2活跃月份
2讨论串
1,415被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Standard accuracy metrics indicate that reading comprehension systems are making rapid progress, but the extent to which these systems truly understand language remains unclear. To reward systems with real language understanding abilities, we propose an adversarial evaluation scheme for the Stanford Question Answering Dataset (SQuAD). Our method tests whether systems can answer questions about paragraphs that contain adversarially inserted sentences, which are automatically generated to distract computer systems without changing the correct answer or misleading humans. In this adversarial setting, the accuracy of sixteen published models drops from an average of $75\%$ F1 score to $36\%$; when the adversary is allowed to add ungrammatical sequences of words, average accuracy on four models decreases further to $7\%$. We hope our insights will motivate the development of new models that understand language more precisely.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论sanxiyn · 2017-09

“In "Adversarial Examples for Evaluating Reading Comprehension Systems" https://arxiv.org/abs/1707.07328, it was found that adding a single distracting sentence can lower F1 score of BiDAF (which is used in demo here) from 75.5% to 34.3% on SQuAD. In comparison, human performance goes from 92.6% to 89.2%.”

AllenNLP – An open-source NLP research library, built on PyTorch原评论 ↗
评论rspeer · 2017-09

“Heavily trained on SQuAD questions. There are lots of models out there that are very good at recognizing SQuAD questions, and reverse-engineering the predictable ways that the Turkers who wrote the questions pulled the information out of the paragraph -- allowing them to answer the question without ever understanding it. https://arxiv.org/abs/1707.07328”

AllenNLP – An open-source NLP research library, built on PyTorch原评论 ↗
评论pegasos1 · 2018-01

“This is clickbait. Unless models are invariant to adversarial examples in SQuAD such as those described here: https://arxiv.org/abs/1707.07328, models doing really well on SQuAD doesn't mean a ton.”

Alibaba neural network defeats human in global reading test原评论 ↗