按年被提到
20192019 · 1 次2026
大模型与语言模型›词向量与预训练模型·arXiv:1909.11687 · cs.CL

Extremely Small BERT Models from Mixed-Vocabulary Training

Sanqiang Zhao, Raghav Gupta, Yang Song, Denny Zhou · 2019
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2019-09-29 · 最近 2023-12-26
2不同的人
2活跃月份
2讨论串
36被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Pretrained language models like BERT have achieved good results on NLP tasks, but are impractical on resource-limited devices due to memory footprint. A large fraction of this footprint comes from the input embeddings with large input vocabulary and embedding dimensions. Existing knowledge distillation methods used for model compression cannot be directly applied to train student models with reduced vocabulary sizes. To this end, we propose a distillation method to align the teacher and student embeddings via mixed-vocabulary training. Our method compresses BERT-LARGE to a task-agnostic model with smaller vocabulary and hidden dimensions, which is an order of magnitude smaller than other distilled BERT models and offers a better size-accuracy trade-off on language understanding benchmarks as well as a practical dialogue task.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论dmezzetti · 2023-12

“For those interested in micro/tiny/small models, a couple more links. https://arxiv.org/abs/1909.11687 https://arxiv.org/abs/2210.07904 I wrote an article on the subject earlier this year - https://neuml.hashnode.dev/train-a-language-model-from-scratch”

EELBERT: Tiny Models Through Dynamic Embeddings原评论 ↗
被提交为 story · 1 次