“About learning, the ERNIE language model has continuous learning meaning it has not the necessity to fully retrain (I believe) Also GPT 3 is obscoleted by order of magnitudes by SMIM https://arxiv.org/abs/2003.02645”
SentenceMIM: A Latent Variable Language Model
SentenceMIM is a probabilistic auto-encoder for language data, trained with Mutual Information Machine (MIM) learning to provide a fixed length representation of variable length language observations (i.e., similar to VAE). Previous attempts to learn VAEs for language data faced challenges due to posterior collapse. MIM learning encourages high mutual information between observations and latent variables, and is robust against posterior collapse. As such, it learns informative representations whose dimension can be an order of magnitude higher than existing language VAEs. Importantly, the SentenceMIM loss has no hyper-parameters, simplifying optimization. We compare sentenceMIM with VAE, and AE on multiple datasets. SentenceMIM yields excellent reconstruction, comparable to AEs, with a rich structured latent space, comparable to VAEs. The structured latent representation is demonstrated with interpolation between sentences of different lengths. We demonstrate the versatility of sentenceMIM by utilizing a trained model for question-answering and transfer learning, without fine-tuning, outperforming VAE and AE with similar architectures.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“GPT 3 is obscoleted by order of magnitudes. SMIM has achieved 4.6 of perplexity vs 20 for GPT 3 with and with a thousand less parameters https://arxiv.org/abs/2003.02645 This is the breakthrough of the year and will be silent until the few nerds like me propagate the news to the mainstream”
“Meanwhile, a revolutionary paper that brought for the first successful time a new paradigm to NLP (latent variational autoencoders) and that destroy GPT 3 on text perplexity on the Pen treebank (4.6 vs 20) and with order of magnitudes less parameters is talked about nowhere on the web... https://arxiv.org/abs/2003.02645v2”
- SentenceMIM: A Latent Variable Language Model ↗telotortium · 2020-06-11 · 1 分