按年被提到
20202020 · 4 次2026
大模型与语言模型›词向量与预训练模型·arXiv:2003.02645 · cs.CL

SentenceMIM: A Latent Variable Language Model

Micha Livne, Kevin Swersky, David J. Fleet · 2020
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-05-29 · 最近 2020-06-11
2不同的人
2活跃月份
3讨论串
2被引(OpenAlex)
arXiv 摘要(候选层不写导读)

SentenceMIM is a probabilistic auto-encoder for language data, trained with Mutual Information Machine (MIM) learning to provide a fixed length representation of variable length language observations (i.e., similar to VAE). Previous attempts to learn VAEs for language data faced challenges due to posterior collapse. MIM learning encourages high mutual information between observations and latent variables, and is robust against posterior collapse. As such, it learns informative representations whose dimension can be an order of magnitude higher than existing language VAEs. Importantly, the SentenceMIM loss has no hyper-parameters, simplifying optimization. We compare sentenceMIM with VAE, and AE on multiple datasets. SentenceMIM yields excellent reconstruction, comparable to AEs, with a rich structured latent space, comparable to VAEs. The structured latent representation is demonstrated with interpolation between sentences of different lengths. We demonstrate the versatility of sentenceMIM by utilizing a trained model for question-answering and transfer learning, without fine-tuning, outperforming VAE and AE with similar architectures.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论The_rationalist · 2020-05

“About learning, the ERNIE language model has continuous learning meaning it has not the necessity to fully retrain (I believe) Also GPT 3 is obscoleted by order of magnitudes by SMIM https://arxiv.org/abs/2003.02645”

GPT-3: Language Models Are Few-Shot Learners原评论 ↗
评论The_rationalist · 2020-05

“GPT 3 is obscoleted by order of magnitudes. SMIM has achieved 4.6 of perplexity vs 20 for GPT 3 with and with a thousand less parameters https://arxiv.org/abs/2003.02645 This is the breakthrough of the year and will be silent until the few nerds like me propagate the news to the mainstream”

GPT-3: Language Models Are Few-Shot Learners原评论 ↗
评论The_rationalist · 2020-05

“Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2”

GPT-3: A Disappointing Paper?原评论 ↗
被提交为 story · 1 次