按年被提到
20212021 · 2 次2026
大模型与语言模型›量化、解码与分词·arXiv:2110.13711 · cs.LG

Hierarchical Transformers Are More Efficient Language Models

Piotr Nawrot, Szymon Tworkowski, Michał Tyrolski, Łukasz Kaiser, Yuhuai Wu, Christian Szegedy, Henryk Michalewski · 2021
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2021-11-04 · 最近 2023-03-21
3不同的人
2活跃月份
3讨论串
41被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Transformer models yield impressive results on many NLP and sequence modeling tasks. Remarkably, Transformers can handle long sequences which allows them to produce long coherent outputs: full paragraphs produced by GPT-3 or well-structured images produced by DALL-E. These large language models are impressive but also very inefficient and costly, which limits their applications and accessibility. We postulate that having an explicit hierarchical architecture is the key to Transformers that efficiently handle long sequences. To verify this claim, we first study different ways to downsample and upsample activations in Transformers so as to make them hierarchical. We use the best performing upsampling and downsampling layers to create Hourglass - a hierarchical Transformer language model. Hourglass improves upon the Transformer baseline given the same amount of computation and can yield the same results as Transformers more efficiently. In particular, Hourglass sets new state-of-the-art for Transformer models on the ImageNet32 generation task and improves language modeling efficiency on the widely studied enwik8 benchmark.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论robbedpeter · 2021-11

“The hierarchical transformers variants are uncovering some possible optimizations that are similar to the ideas of Thousand Brains - https://arxiv.org/abs/2110.13711 The attention mechanisms, in conjunction with autoencoding, create a rough approximation of what grid cells accomplish, but transformers are still a feedforward architecture. Thanks to Moore…”

Transformers from Scratch原评论 ↗
评论korec123 · 2023-03

“There is another line of work for efficient Transformers that wasn't mentioned in the paper, i.e., Adaptive Computation on the sequence level, which pools similar tokens which are easy to predict, therefore reducing the complexity of a Transformer layer. Examples [#1](https://arxiv.org/pdf/2211.09761.pdf) [#2](https://arxiv.org/pdf/2103.06874.pdf) [#3](ht…”

CoLT5: Faster Long-Range Transformers With Conditional Computation原评论 ↗
被提交为 story · 1 次