按年被提到
20192020 · 1 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:1904.02679 · cs.HC

Visualizing Attention in Transformer-Based Language Representation Models

Jesse Vig · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-07-28 · 最近 2023-05-24
2不同的人
2活跃月份
2讨论串
88被引(OpenAlex)
arXiv 摘要(候选层不写导读)

We present an open-source tool for visualizing multi-head self-attention in Transformer-based language representation models. The tool extends earlier work by visualizing attention at three levels of granularity: the attention-head level, the model level, and the neuron level. We describe how each of these views can help to interpret the model, and we demonstrate the tool on the BERT model and the OpenAI GPT-2 model. We also present three use cases for analyzing GPT-2: detecting model bias, identifying recurring patterns, and linking neurons to model behavior.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论ypcx · 2020-07

“Yes this seems like an early work in progress, compared to Jay's previous Transformer articles. In addition to your link, I've found a really good Transformer explanation here (backed by a Github repo w/ lively Issues talk): http://www.peterbloem.nl/blog/transformers Additionally, there's a paper on visualizing self-attention: https://arxiv.org/pdf/1904…”

How GPT3 Works – Visualizations and Animations原评论 ↗
评论visarga · 2023-05

“The transformer doesn't have the nice pyramid shape of CNNs, but it still needs multiple layers. There have been papers showing non-trivial interactions between successive layers, forming more complex circuits. https://transformer-circuits.pub/2021/framework/index.html (warning, advanced difficulty) The Q an K matrices learn how to relate tokens. Each o…”

Why the Original Transformer LLM Figure Is Wrong, and Other Interesting Tidbits原评论 ↗