“Definitions are not enough to fully capture the meaning of a word. In order to do that you need full language modelling and to ground words into other sensory modalities, plus the word in relation to actions taken in various situations when the word was used. GPT-2 (of recent OpenAI fame) uses 1.5 billion parameters and, though capable of interesting res…”
Character-Level Language Modeling with Deeper Self-Attention
LSTMs and other RNN variants have shown strong performance on character-level language modeling. These models are typically trained using truncated backpropagation through time, and it is common to assume that their success stems from their ability to remember long-term contexts. In this paper, we show that a deep (64-layer) transformer model with fixed context outperforms RNN variants by a large margin, achieving state of the art on two popular benchmarks: 1.13 bits per character on text8 and 1.06 on enwik8. To get good results at this depth, we show that it is important to add auxiliary losses, both at intermediate network layers and intermediate sequence positions.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“It removes you from the list of people who get pwned with a rainbow table from leaked passwords, but that's not much security. English has only got ~1.2 bits of entropy per character. [1] [1] Table 1 in https://arxiv.org/pdf/1808.04444.pdf”