“and then on the BERT structure which does a conceptually similar thing but with a large neural network that can learn more from the same data.”
BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
双向 Transformer 预训练,加一个输出层即可微调多种任务
论文提出 BERT,在所有层中同时以左右上下文为条件,从无标注文本预训练深度双向表示,只需额外加一个输出层微调,就能用于问答、语言推断等多种任务,无需大幅修改任务架构。BERT 在 11 项 NLP 任务上取得当时最优,例如 GLUE 80.5%、MultiNLI 86.7%、SQuAD v1.1 F1 93.2、SQuAD v2.0 F1 83.1。
BERT 常被当作「BERT 原始论文」引用,用于说明预训练加迁移、BERT 与 GPT 的差别,以及 Google 在 NLP 上的早期贡献。
学习路径
读这篇之前,按顺序读书架上的 6 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先从词向量一路读到上下文化表示、预训练微调和 Transformer,就能理解 BERT 为什么用双向 Transformer 加掩码语言模型来做预训练。
- 1
无监督预训练得到可迁移表示的最早成功例子
读CBOW 与 Skip-gram 的目标;理解静态词向量为什么不够
- 2
- 3
BERT 沿用的“预训练后整体微调”做法
读微调流程与逐层解冻、判别式学习率;对照 BERT 的微调
- 4
- 5
BERT 的 WordPiece 子词分词来自这个系统
读WordPiece 分词部分;理解为何用子词处理未登录词
- 6
BookCorpus 预训练语料的来源
读只需了解 BookCorpus 数据的构成;不必读视觉对齐部分
- 7BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding
目标论文:双向 Transformer 加掩码语言模型预训练
读第 3 节的 MLM 与下一句预测任务、输入表示,第 4 节各任务微调方式
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“All of the parameters of BERT and W are fine-tuned jointly to maximize the log-probability of the correct label.”
“Google's BERT paper does sentence-level pretraining and transfer to get state of the art results on a bunch of problems: BERT”
“BERT used GPT's architecture but trained in a different way.”
“OK, now I've clicked the correct random link and got the earlier paper: BERT”
“And they claim to be bery flexible, too[2].”
看全部 20 条 →
“If you compare to BERT paper BERT training data for”
“original BERT paper:BERT”
“BERT, which stands for Bidirectional Encoder Representations from Transformers.”
“they have led, and continue to lead, much of the pioneering research in NLP and neural IR for the past decade [3][4][5].”
“You mean NLP field changing models from Google like BERT [1]?”
“I posted a list of papers on twitter, and will be posting a summary for each of them as well.”
“2018 bert: BERT”
“For people who are not aware, Jacob Devlin isn't just some random Google engineer, he was one of the authors of the original BERT paper.”
“the paper [1] mentions that the pretraining took 4 days on 16 TPU chips for the BERT-Base model”
“Is it this one? "BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding":BERT”
“BERT: BERT”
“OpenAI was most certainly not the first organization to develop (or distribute) an LLM model:”
“The term 'GPT' was first used in the BERT paper to refer to Generic Pre-trained Transformers.”
“BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding: BERT”