按年被提到
20172018 · 1 次2026
大模型与语言模型›序列模型与注意力机制·arXiv:1708.02182 · cs.CL

Regularizing and Optimizing LSTM Language Models

Stephen Merity, Nitish Shirish Keskar, Richard Socher · 2017
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2018-05-04 · 最近 2024-07-24
2不同的人
2活跃月份
2讨论串
697被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Recurrent neural networks (RNNs), such as long short-term memory networks (LSTMs), serve as a fundamental building block for many sequence learning tasks, including machine translation, language modeling, and question answering. In this paper, we consider the specific problem of word-level language modeling and investigate strategies for regularizing and optimizing LSTM-based models. We propose the weight-dropped LSTM which uses DropConnect on hidden-to-hidden weights as a form of recurrent regularization. Further, we introduce NT-ASGD, a variant of the averaged stochastic gradient method, wherein the averaging trigger is determined using a non-monotonic condition as opposed to being tuned by the user. Using these and other regularization strategies, we achieve state-of-the-art word level perplexities on two data sets: 57.3 on Penn Treebank and 65.8 on WikiText-2. In exploring the effectiveness of a neural cache in conjunction with our proposed model, we achieve an even lower state-of-the-art perplexity of 52.8 on Penn Treebank and 52.0 on WikiText-2.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论cs702 · 2018-05

“This is the kind of problem for which LSTM RNNs -- and more recently, fully-attention-based deep neural nets -- produce state-of-the-art results. I wonder if the author ever tried using, say, an AWD LSTM RNN[a] or a Transformer-like model[b] for this task. Using an RNN or an attention model for this would eliminate the need for feature engineering such …”

Extracting Structured Data from Recipes Using Conditional Random Fields (2015)原评论 ↗
评论jszymborski · 2024-07

“Salesforce has long been involved in publishing quality NLP papers, especially during Stephen Merity's tenure. Smerity's papers are some of my favourite. Check out https://ar5iv.labs.arxiv.org/html/1708.02182 And my all-time favourite https://ar5iv.labs.arxiv.org/html/1911.11423”

A multimodal dataset with one trillion tokens原评论 ↗