“truly revolutionary: It basically lets you create a very small”
LoRA: Low-Rank Adaptation of Large Language Models
冻结预训练权重,只训练低秩矩阵来微调大模型
随着预训练模型变大,重新训练全部参数的全量微调越来越不可行。LoRA 冻结预训练权重,并在 Transformer 的每一层注入可训练的秩分解矩阵,大幅减少下游任务的可训练参数。相比用 Adam 微调的 GPT-3 175B,LoRA 可把可训练参数减少 10,000 倍、GPU 显存需求减少 3 倍,在 RoBERTa、DeBERTa、GPT-2 和 GPT-3 上的模型质量与全量微调相当或更好,训练吞吐更高,且不像 adapter 那样增加推理延迟。
HN 上的人把它当作让个人在消费级 GPU 上微调大模型成为可能的关键方法,常引用它解释低秩更新如何合并回基础权重,并强调其效果可与全量微调持平。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径先建立 Transformer 结构和预训练-微调范式,再看 GPT-3 带来的全量微调成本问题,最后读 LoRA 如何只训练低秩矩阵。
- 1
LoRA 要注入低秩矩阵的正是 Transformer 里的注意力投影矩阵。
读多头自注意力里 Q、K、V、输出投影的权重矩阵;LoRA 主要改这些矩阵,其余可略读。
- 2
BERT 确立预训练后全参数微调的范式,LoRA 要解决的正是这种范式的成本。
读预训练加下游微调的流程:每个任务都要保存一份完整模型副本;实验细节可跳过。
- 3
LoRA 的实验以 RoBERTa 为基座之一,和全量微调做对比。
读RoBERTa 在 GLUE 上的训练配置,便于读懂 LoRA 实验表里的全量微调基线。
- 4
1750 亿参数的 GPT-3 让每个任务一份微调副本变得不可行,这是 LoRA 的动机。
读模型规模与参数量部分,以及少样本提示作为不微调的替代方案;LoRA 实验也在 GPT-3 175B 上做。
- 5LoRA: Low-Rank Adaptation of Large Language Models
目标论文:冻结预训练权重,只训练低秩分解矩阵。
读第 4 节的低秩更新 ΔW = BA 与初始化、可合并回原权重所以无推理延迟;再看秩 r 的实验分析。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“and read up on LoRA That should get you started.”
“For detailed explanation I recommend the paper.”
“You should look into LoRA”
“Be aware that LoRA performs on-par or better than fine-tuning in model quality if trained correctly as the paper shows: LoRA”
“LoRA, Low-rank Adaptation of LLMs, consists in adapting the weights of a big neural network to a target task”
看全部 18 条 →
“the recent references to LoRA stands for Low-Rank Adaptation [1], not Low-Resource Adapters[2]”
“They address prompt tuning's issues in the paper:”
“See the abstract of LoRA”
“The hit seems to be in energy/cpu not time since the W0 computation is in parallel with the BAx. (My assumption based on the latency claims in paper.)”
“The LoRA paper clearly states the performance of the method”
“Open-source LLM projects have largely solved this using Low-Rank Adaptation of Large Language Models (LoRA): LoRA”
“Somewhat yes. See "LoRA": LoRA”
“This seems to be an order of magnitude better than LoRa ([LINK] not LoRA).”
“The idea from LoRA[1] is to take pre-trained, dense weights W_0 for a model and adapt them to new training data by using the weights W = W_0 + BA for inference.”
“The LoRA paper[1] specifically says that all of the pretrained weights remain frozen.”
“the equation for the forward pass from the original paper[1]:”
“once the industry matures a bit we'll go back to things like LoRA[1] and control vectors[2][3].”
- LORA: Low-Rank Adaptation of Large Language Models ↗LukeEF · 2023-05-05 · 1 分
- LoRA: Low-Rank Adaptation of Large Language Models (2021) ↗tosh · 2026-10-15 · 1 分