按年被提到
20202020 · 1 次2026
大模型与语言模型›量化、解码与分词·arXiv:2003.10555 · cs.CL

ELECTRA: Pre-training Text Encoders as Discriminators Rather Than Generators

Kevin Clark, Minh-Thang Luong, Quoc V. Le, Christopher D. Manning · 2020
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2020-11-28 · 最近 2024-07-16
2不同的人
2活跃月份
2讨论串
2,437被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Masked language modeling (MLM) pre-training methods such as BERT corrupt the input by replacing some tokens with [MASK] and then train a model to reconstruct the original tokens. While they produce good results when transferred to downstream NLP tasks, they generally require large amounts of compute to be effective. As an alternative, we propose a more sample-efficient pre-training task called replaced token detection. Instead of masking the input, our approach corrupts it by replacing some tokens with plausible alternatives sampled from a small generator network. Then, instead of training a model that predicts the original identities of the corrupted tokens, we train a discriminative model that predicts whether each token in the corrupted input was replaced by a generator sample or not. Thorough experiments demonstrate this new pre-training task is more efficient than MLM because the task is defined over all input tokens rather than just the small subset that was masked out. As a result, the contextual representations learned by our approach substantially outperform the ones learned by BERT given the same model size, data, and compute. The gains are particularly strong for small models; for example, we train a model on one GPU for 4 days that outperforms GPT (trained using 30x more compute) on the GLUE natural language understanding benchmark. Our approach also works well at scale, where it performs comparably to RoBERTa and XLNet while using less than 1/4 of their compute and outperforms them when using the same amount of compute.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论qayxc · 2020-11

“In this particular case it's excusable as English is the Lingua Franca of science today. Used to be Latin, then French and German, now it's English. No big deal IMO. In fact I kind of like the way this is going since it represents a fantastic opportunity for NL researchers to stand out simply by publishing research and corpora focused exclusively on low-…”

TLDR: Extreme Summarization of Scientific Documents原评论 ↗
被提交为 story · 1 次