按年被提到
20192020 · 1 次2026
视觉、语音与图像生成›ViT 与视觉新架构·arXiv:1904.10509 · cs.LG

Generating Long Sequences with Sparse Transformers

Rewon Child, Scott Gray, Alec Radford, Ilya Sutskever · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-03-28 · 最近 2024-01-16
2不同的人
2活跃月份
2讨论串
1,607被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Transformers are powerful sequence models, but require time and memory that grows quadratically with the sequence length. In this paper we introduce sparse factorizations of the attention matrix which reduce this to $O(n \sqrt{n})$. We also introduce a) a variation on architecture and initialization to train deeper networks, b) the recomputation of attention matrices to save memory, and c) fast attention kernels for training. We call networks with these changes Sparse Transformers, and show they can model sequences tens of thousands of timesteps long using hundreds of layers. We use the same architecture to model images, audio, and text from raw bytes, setting a new state of the art for density modeling of Enwik8, CIFAR-10, and ImageNet-64. We generate unconditional samples that demonstrate global coherence and great diversity, and show it is possible in principle to use self-attention to model sequences of length one million or more.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论p1esk · 2020-03

“Interesting you mentioned Makridakis competitions. There's one going on right now on Kaggle, and the current leader believes a NN will be the winning model: https://www.kaggle.com/c/m5-forecasting-accuracy/discussion/138881 More generally, it seems that time series forecasting so far has mostly attracted statisticians with little DL experience [1]. Now t…”

A Survey of Deep Learning for Scientific Discovery原评论 ↗
被提交为 story · 1 次