“It's in the DeepSeek V3 paper, not the R1 paper.”
DeepSeek-V3 Technical Report
671B 参数 MoE 模型,完整训练仅需 2.788M H800 GPU 小时
DeepSeek-V3 是总参数 671B、每个 token 激活 37B 的混合专家(MoE)语言模型,采用经 DeepSeek-V2 验证的多头潜在注意力(MLA)和 DeepSeekMoE 架构,并首创无辅助损失的负载均衡策略,同时使用多 token 预测训练目标。模型在 14.8 万亿 token 上预训练,再经监督微调与强化学习。评测显示它超过其他开源模型、与领先闭源模型相当;完整训练只需 2.788M H800 GPU 小时,且训练过程稳定,没有不可恢复的损失尖峰或回滚。
评论者多是为查证而引用它:约 560 万美元(2.788M H800 GPU 小时)的训练成本说法、DualPipe 与通信优化、PTX 的实际使用范围,以及 256 个专家、每层路由 8 个的 MoE 结构细节。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Their model is open and they published paper describing it DeepSeek-V3 Technical Report”
“They also did bandwidth scaling to handle work around the nerfed H800 interconnects.”
“The DeepSeek v3 model had a net training cost of >$5m for the final training run, the paper lists over 100 authors[1], meaning highly-paid engineers.”
“There are two papers: DeepSeek-V3 Technical Report for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and [OTHER_PAPER] for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.”
“used by the model defined in MIT-licensed code[0] and described in a technical report[1].”
看全部 20 条 →
“Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even?”
“The DeepSeek paper even gives a breakdown:”
“Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1]”
“Their paper [1] only mentions using PTX in a few areas to optimize data transfer operations so they don't blow up the L2 cache.”
“For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1]”
“DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer DeepSeek-V3 Technical Report”
“The Deepseek v3 paper details a quantisation method of scaling after matmul but before accumulation to improve precision”
“DeepSeek-V3 costs only 2.788M GPU hours for its full training.”
“Specifically, we employ customized PTX (Parallel Thread Execution) instructions”
“The DeepSeek v3 paper claims to have trained from scratch for ~$5.5m: DeepSeek-V3 Technical Report”
“Uh yes? Deepseek explicitly said they used H800s [1].”
“For example, DeepSeek-V3 [1], still a relatively large model:”
“That's deepseek using FP8 live weight copy + FP32 master + FP32 grad + BF16 moments = 13 bytes per parameter.”
“DeepSeek V3 shipped MTP in a product first, in 2024:”