“I was impressed by the detail and openness of Meta”
The Llama 3 Herd of Models
Meta 的 Llama 3 技术报告:最大 405B 稠密 Transformer,原生多语言与工具使用
介绍 Meta 的 Llama 3 基础模型群,原生支持多语言、编程、推理和工具使用;最大模型是 405B 参数的稠密 Transformer,上下文窗口最长 128K 词元。大量评测显示它在众多任务上与 GPT-4 等领先模型质量相当。论文公开发布预训练和后训练版本的 405B 模型以及 Llama Guard 3,并以组合式方法加入图像、视频和语音能力,在这些识别任务上与当时最好水平相当,但相应模型尚未广泛发布。
HN 上的人称赞它细节详尽、公开度高,常引用其中关于训练集群、编码微调、事实性训练和合成数据的细节;也有人批评它没有说明数据来源。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check out the technical report of Llama 3 for instance, with nice figures on how scaling up model training does increase performance on intelligence tests (might as well call that intelligence): The Llama 3 Herd of Models”
“those papers notably do not say where the data comes from.”
“Llama 3 paper [0], section 4.3.1 mentions what sort of work is needed.”
“I think we need more fundamental breakthroughs than attention-based neural networks and all the tricks that have gone into say a Llama3. [1]”
“Facebook did a technical paper where they described their training cluster and the sheer amount of complexity is staggering.”
看全部 9 条 →
“Meta described a clever way in their paper on training Llama3 [1] (in section about factuality).”
“Meta synthetically generated lots of PHP from Python for Llama 3 for training purposes.”
“1e21 flops is hilariously wrong. for reference the llama 3 8B model used 10 times that many flops.”
- The Llama 3 Herd of Models ↗GaggiX · 2024-08-01 · 1 分
- The Llama 3 Herd of Models ↗cloudsql · 2024-12-09 · 1 分