按年被提到
20232023 · 25 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2302.13971 · cs.CL

LLaMA: Open and Efficient Foundation Language Models

LLaMA:开放且高效的基础语言模型
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin 等 14 人 · 2023

LLaMA:只用公开数据训练的 7B 至 65B 开放基础语言模型

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 26 人首次 2023-03-06 · 最近 2025-02-07
26不同的人
10活跃月份
25讨论串
11,665被引(OpenAlex)

论文介绍 LLaMA,一组参数从 70 亿到 650 亿的基础语言模型,在数万亿 token 上训练,并表明只用公开可得的数据集就能训练出最先进的模型,而无需依赖专有且无法获取的数据。其中 LLaMA-13B 在多数基准上超过 GPT-3(175B),LLaMA-65B 与最佳模型 Chinchilla-70B 和 PaLM-540B 相当,所有模型都向研究社区发布。

HN 为什么提它

评论者主要引用它的细节:训练数据来源(含 books3)、与 GPT-3 等模型的基准对比、大模型训练不足的发现,以及训练算力(如 65B 用 2048 张 A100 训练 21 天)和 token 数量。

方法难度 3 / 5约 60 分钟有时效性
读之前最好知道
1Transformer 架构
2语言模型预训练
3缩放定律(scaling laws)与 Chinchilla
4大模型评测基准
5训练数据混合与 token 化

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐simonw · 2024-03

“The most detailed answer to that I've seen is the original LLaMA paper”

推荐 LLaMA 论文,称它最详细地说明了训练数据来源
DBRX: A new open LLM原评论 ↗
推荐michaelt · 2024-06

“Then, you should check out papers like LLaMA and [OTHER_PAPER]”

推荐看 LLaMA 论文,表 1 披露了数据来源,包括 books3
How Meta trains large language models at scale原评论 ↗
引用zavrel · 2023-03

“Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models LLaMA”

希望能托管运行 LLaMA 这个模型
The Most Cited AI Papers in 2022原评论 ↗
引用doctoboggan · 2023-03

“According to Meta's benchmarking[0] it is comparable on many metrics.”

引用 Meta 的基准测试:LLaMA 在许多指标上与对比模型相当
LLaMA-7B in Pure C++ with full Apple Silicon support原评论 ↗
引用v64 · 2023-03

“One finding in the LLaMA paper [1] is that our current large models are undertrained.”

引用 LLaMA 论文的发现:现有大模型训练不足
ChatGPT's API Is So Good and Cheap, It Makes Most Text Generating AI Obsolete原评论 ↗
引用sitic · 2023-03

“The LLaMA paper contradicts this view:”

用 LLaMA 论文反驳前面的观点:7B 在超过 1T token 后仍在提升
The genie escapes: Stanford copies the ChatGPT AI for less than $600原评论 ↗
看全部 20 条 →
引用rnosov · 2023-03

“There are benchmarks in the original LLaMA paper[1].”

指出 LLaMA 原论文里有基准测试可查
RWKV RNN: Better than ChatGPT?原评论 ↗
引用gpm · 2023-03

“Edit: The huggingface page has 0-shot benchmarks which you can compare against the llama paper”

拿 LLaMA 论文的数字对照 Cerebras-GPT 的 0-shot 基准
Cerebras-GPT: A Family of Open, Compute-Efficient, Large Language Models原评论 ↗
引用JW_00000 · 2023-03

“The LLaMA paper [1] (Meta's model) contains details about what they trained it on.”

指出 LLaMA 论文详述了模型的训练数据
引用imranq · 2023-03

“Looking back at the Llama paper, many of the results are comparable between llama and RWKV.”

回看 LLaMA 论文,称 LLaMA 与 RWKV 的许多结果相当
The RWKV language model: An RNN with the advantages of a transformer原评论 ↗
引用sp332 · 2023-04

“Check out the graph on page 3 of this PDF: LLaMA The 33B model started beating the 7B when it had been trained on only 1/3 as much data.”

指向论文第 3 页的图:33B 只训练了 1/3 数据就开始超过 7B
Llama.cpp 30B runs with only 6GB of RAM now原评论 ↗
引用yreg · 2023-04

“Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0].”

引用它说明 LLaMA 的表现最接近 GPT-3
An LLM playground you can run on your laptop原评论 ↗
引用lhl · 2023-04

“From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals.”

引用 Meta 基准:LLaMA 33B 胜过 GPT-3 175B 等,并在不少评测上追平 PaLM 540B
The LLama Effect: Leak Sparked a Series of Open Source Alternatives to ChatGPT原评论 ↗
引用eiz · 2023-04

“LLaMA table 15. 1770394 A100-80GB hours to train the entire model suite”

引用表 15:训练整套模型共用 1770394 个 A100-80GB 小时
RedPajama: Reproduction of Llama with Friendly License (Stanford, MILA Quebec)原评论 ↗
引用nl · 2023-04

“LLama 65B used 2048 80G A100s for 21 days:”

引用 LLaMA 65B 用 2048 块 A100 训练 21 天的数据
RedPajama: Reproduction of Llama with Friendly License (Stanford, MILA Quebec)原评论 ↗
引用YetAnotherNick · 2023-05

“Hellaswag accuracy for llama 7b is 0.76[1], but it is written 0.56 in the repo.”

指出 LLaMA 7B 的 HellaSwag 准确率论文中是 0.76,仓库里却写成 0.56
OpenLLaMA: An Open Reproduction of LLaMA原评论 ↗
引用capableweb · 2023-05

“Which in turn has the following as the first link: LLaMA”

仅把它作为开放基础模型的链接提到
How to run Llama 13B with a 6GB graphics card原评论 ↗
引用ppaattrriicckk · 2023-05

“looking at the number of tokens LLaMA[1] was trained on (1,400B tokens)”

引用 LLaMA 的训练 token 数(1,400B)
Intel Announces Aurora GenAI, ChatGPT Competitor with 1T Parameters原评论 ↗
引用londons_explore · 2023-06

“Instead, I think a small team of a few researchers made LLaMA and published a paper about it[1], and then realised to get any citations for their paper they'd need to let other researchers have the weights.”

推测一小队研究者发了论文,为获得引用才向其他研究者开放权重
Senators send letter questioning Mark Zuckerberg over Meta’s LLaMA leak原评论 ↗
引用pythux · 2023-06

“Table 15 from LLaMA paper[1] has some insightful figures: it took 135,168 GPU hours to train the 13B version and a bit more than 1M GPU hours for the 65B version.”

引用表 15:13B 约需 135,168 GPU 小时,65B 略多于 1M
OpenLLaMA 13B Released原评论 ↗