按年被提到
20182019 · 1 次2026
大模型与语言模型›序列模型与注意力机制·arXiv:1808.04444 · cs.CL

Character-Level Language Modeling with Deeper Self-Attention

Rami Al-Rfou, Dokook Choe, Noah Constant, Mandy Guo, Llion Jones · 2018
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-03-08 · 最近 2020-11-08
2不同的人
2活跃月份
2讨论串
71被引(OpenAlex)
arXiv 摘要(候选层不写导读)

LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论visarga · 2019-03

“Definitions are not enough to fully capture the meaning of a word. In order to do that you need full language modelling and to ground words into other sensory modalities, plus the word in relation to actions taken in various situations when the word was used. GPT-2 (of recent OpenAI fame) uses 1.5 billion parameters and, though capable of interesting res…”

What's the minimum number of words you'd need to define all other words? (2012)原评论 ↗
评论rmrfstar · 2020-11

“It removes you from the list of people who get pwned with a rainbow table from leaked passwords, but that's not much security. English has only got ~1.2 bits of entropy per character. [1] [1] Table 1 in https://arxiv.org/pdf/1808.04444.pdf”

Finally: A usable and secure password policy backed by science原评论 ↗