“The most detailed answer to that I've seen is the original LLaMA paper”
LLaMA: Open and Efficient Foundation Language Models
LLaMA:只用公开数据训练的 7B 至 65B 开放基础语言模型
论文介绍 LLaMA,一组参数从 70 亿到 650 亿的基础语言模型,在数万亿 token 上训练,并表明只用公开可得的数据集就能训练出最先进的模型,而无需依赖专有且无法获取的数据。其中 LLaMA-13B 在多数基准上超过 GPT-3(175B),LLaMA-65B 与最佳模型 Chinchilla-70B 和 PaLM-540B 相当,所有模型都向研究社区发布。
评论者主要引用它的细节:训练数据来源(含 books3)、与 GPT-3 等模型的基准对比、大模型训练不足的发现,以及训练算力(如 65B 用 2048 张 A100 训练 21 天)和 token 数量。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Then, you should check out papers like LLaMA and [OTHER_PAPER]”
“Would be wonderful to use this host this model LLaMA: Open and Efficient Foundation Language Models LLaMA”
“According to Meta's benchmarking[0] it is comparable on many metrics.”
“One finding in the LLaMA paper [1] is that our current large models are undertrained.”
“The LLaMA paper contradicts this view:”
看全部 20 条 →
“There are benchmarks in the original LLaMA paper[1].”
“Edit: The huggingface page has 0-shot benchmarks which you can compare against the llama paper”
“The LLaMA paper [1] (Meta's model) contains details about what they trained it on.”
“Looking back at the Llama paper, many of the results are comparable between llama and RWKV.”
“Check out the graph on page 3 of this PDF: LLaMA The 33B model started beating the 7B when it had been trained on only 1/3 as much data.”
“Incredibly difficult to measure it, but it seems llama is closest to GPT-3[0].”
“From Meta's benchmarking [1] LLaMA 33B outperforms GPT-3 175B, Gopher 280B, Chinchilla 70B and even matches PaLM 540B on a bunch of common evals.”
“LLaMA table 15. 1770394 A100-80GB hours to train the entire model suite”
“LLama 65B used 2048 80G A100s for 21 days:”
“Hellaswag accuracy for llama 7b is 0.76[1], but it is written 0.56 in the repo.”
“Which in turn has the following as the first link: LLaMA”
“looking at the number of tokens LLaMA[1] was trained on (1,400B tokens)”
“Instead, I think a small team of a few researchers made LLaMA and published a paper about it[1], and then realised to get any citations for their paper they'd need to let other researchers have the weights.”
“Table 15 from LLaMA paper[1] has some insightful figures: it took 135,168 GPU hours to train the 13B version and a bit more than 1M GPU hours for the 65B version.”