按年被提到
20192020 · 1 次2026
大模型与语言模型›可解释性与表征研究·arXiv:1908.04626 · cs.CL

Attention is not not Explanation

Sarah Wiegreffe, Yuval Pinter · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-12-08 · 最近 2026-01-08
2不同的人
2活跃月份
2讨论串
1,050被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Attention mechanisms play a central role in NLP systems, especially within recurrent neural network (RNN) models. Recently, there has been increasing interest in whether or not the intermediate representations offered by these modules may be used to explain the reasoning for a model's prediction, and consequently reach insights regarding the model's decision-making process. A recent paper claims that `Attention is not Explanation' (Jain and Wallace, 2019). We challenge many of the assumptions underlying this work, arguing that such a claim depends on one's definition of explanation, and that testing it needs to take into account all elements of the model, using a rigorous experimental design. We propose four alternative tests to determine when/whether attention can be used as explanation: a simple uniform-weights baseline; a variance calibration based on multiple random seed runs; a diagnostic framework using frozen weights from pretrained models; and an end-to-end adversarial attention training protocol. Each allows for meaningful interpretation of attention mechanisms in RNN models. We show that even when reliable adversarial distributions can be found, they don't perform well on the simple diagnostic, indicating that prior work does not disprove the usefulness of attention mechanisms for explainability.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论screye · 2020-12

“The pair of these papers: (Don't read them in full.) 1.Attention is not explanation (https://arxiv.org/abs/1902.10186) 2.Attention is not not Explanation (https://arxiv.org/abs/1908.04626) Goes to show the complete lack of agreement between researchers in the explainability space. Most popular packages (allen NLP, google LIT, Captum) use saliency based…”

Ask HN: What's the best paper you've read in 2020?原评论 ↗
评论D-Machine · 2026-01

“Read the third link / review paper, it is not at all the case that all attention is based on QKV projections. Your terms "sensitivity", "visibility", and "important" are too vague and lack any clear mathematical meaning, so IMO add nothing to any understanding. "Important" also seems factually wrong, given these layers are stacked, so later weights and o…”

The Q, K, V Matrices原评论 ↗