按年被提到
20242025 · 20 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2412.19437 · cs.CL

DeepSeek-V3 Technical Report

DeepSeek-V3 技术报告
DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai 等 200 人 · 2024

671B 参数 MoE 模型,完整训练仅需 2.788M H800 GPU 小时

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 25 人首次 2025-01-27 · 最近 2026-06-21
25不同的人
11活跃月份
29讨论串
2,509被引(OpenAlex)

DeepSeek-V3 是总参数 671B、每个 token 激活 37B 的混合专家(MoE)语言模型,采用经 DeepSeek-V2 验证的多头潜在注意力(MLA)和 DeepSeekMoE 架构,并首创无辅助损失的负载均衡策略,同时使用多 token 预测训练目标。模型在 14.8 万亿 token 上预训练,再经监督微调与强化学习。评测显示它超过其他开源模型、与领先闭源模型相当;完整训练只需 2.788M H800 GPU 小时,且训练过程稳定,没有不可恢复的损失尖峰或回滚。

HN 为什么提它

评论者多是为查证而引用它:约 560 万美元(2.788M H800 GPU 小时)的训练成本说法、DualPipe 与通信优化、PTX 的实际使用范围,以及 256 个专家、每层路由 8 个的 MoE 结构细节。

方法难度 4 / 5约 90 分钟经得起时间
读之前最好知道
1混合专家模型(MoE)
2Transformer 与注意力机制
3多头潜在注意力(MLA)
4GPU 并行训练与通信优化

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
引用logicchains · 2025-01

“It's in the DeepSeek V3 paper, not the R1 paper.”

指出 $5.576M 训练成本出自 V3 论文而非 R1 论文
USA restricts Swiss access to AI computer chips原评论 ↗
引用nabla9 · 2025-01

“Their model is open and they published paper describing it DeepSeek-V3 Technical Report”

指出模型开源且论文公开了细节,以此支持其训练成本的可信度
Nvidia’s $589B DeepSeek rout原评论 ↗
引用alecco · 2025-01

“They also did bandwidth scaling to handle work around the nerfed H800 interconnects.”

说明 DeepSeek 用带宽扩展技巧绕开被削弱的 H800 互联
The Illustrated DeepSeek-R1原评论 ↗
引用shenberg · 2025-01

“The DeepSeek v3 model had a net training cost of >$5m for the final training run, the paper lists over 100 authors[1], meaning highly-paid engineers.”

拿论文超过 100 位作者和训练成本说明高薪工程师的投入
Sam Altman said startups with $10M were 'hopeless' competing with OpenAI原评论 ↗
引用btown · 2025-01

“There are two papers: DeepSeek-V3 Technical Report for DeepSeek-V3-Base, which makes claims about 2.8M H800 GPU hours, and [OTHER_PAPER] for DeepSeek-R1, which was based on V3-Base but doesn't seem to quote the cost of its RL steps.”

指出 V3-Base 论文声称用了 2.8M H800 GPU 小时,R1 论文未给出 RL 成本
Berkeley Researchers Replicate DeepSeek R1's Core Tech for Just $30: A Small Mod原评论 ↗
引用Ukv · 2025-01

“used by the model defined in MIT-licensed code[0] and described in a technical report[1].”

指出模型由 MIT 许可代码实现,并在这份技术报告中有描述
Why DeepSeek had to be open source原评论 ↗
看全部 20 条 →
引用fulladder · 2025-01

“Zero evidence that the above statement is true, and weak evidence (authors' claims) that it is false. Have you read their papers even?”

反驳无证据的说法,指出作者自己的论文给出的成本不支持 $1B+ 预算
Exposed DeepSeek database leaking sensitive information, including chat history原评论 ↗
引用amingilani · 2025-02

“The DeepSeek paper even gives a breakdown:”

指出论文给出了训练成本明细:预训练、上下文扩展、后训练
DeepSeek not as disruptive as claimed, firm has 50k GPUs and spent $1.6B原评论 ↗
引用cratermoon · 2025-02

“Thus you get Open AI spending billions while DeepSeek comes long with, shocking, actual understanding of the hardware and how to optimize for it[1]”

拿它说明 DeepSeek 靠理解硬件并做优化而低成本训练
Building a personal, private AI computer on a budget原评论 ↗
引用suresk · 2025-02

“Their paper [1] only mentions using PTX in a few areas to optimize data transfer operations so they don't blow up the L2 cache.”

指出论文只在少数地方用 PTX 优化数据传输,并非取代 CUDA
Introduction to CUDA programming for Python developers原评论 ↗
引用espadrine · 2025-03

“For instance, DeepSeek R1 uses 404 GB in Q4 quantization[0], containing 256 experts of which 8 are routed to[1]”

拿它说明 MoE 有 256 个 experts、每次路由 8 个,因此推理快
Apple unveils new Mac Studio原评论 ↗
引用zamadatix · 2025-04

“DeepSeek V3/R1 are MoE with 256 experts per layer, actively using 1 shared expert and 8 routed experts per layer DeepSeek-V3 Technical Report”

引用其架构细节:MoE 每层 256 个 experts,1 个共享、8 个路由激活
Meta got caught gaming AI benchmarks原评论 ↗
引用ironbound · 2025-04

“The Deepseek v3 paper details a quantisation method of scaling after matmul but before accumulation to improve precision”

指出论文的量化方法在 matmul 后、累加前缩放以提高精度
Lossless LLM compression for efficient GPU inference via dynamic-length float原评论 ↗
引用yorwba · 2025-05

“DeepSeek-V3 costs only 2.788M GPU hours for its full training.”

引用其完整训练仅需 2.788M GPU 小时的成本数字
DeepSeek’s founder is threatening US dominance in AI race原评论 ↗
引用the_panopticon · 2025-09

“Specifically, we employ customized PTX (Parallel Thread Execution) instructions”

引用论文中用定制 PTX 指令提升训练效率的说法
A Gentle Introduction to CUDA PTX原评论 ↗
引用simonw · 2025-12

“The DeepSeek v3 paper claims to have trained from scratch for ~$5.5m: DeepSeek-V3 Technical Report”

指出论文声称从头训练成本约 $5.5M
The Future of Software Development Is Software Developers原评论 ↗
引用cherryteastain · 2026-02

“Uh yes? Deepseek explicitly said they used H800s [1].”

指出论文明确写了用的是 H800
GLM-5: From Vibe Coding to Agentic Engineering原评论 ↗
引用cubefox · 2026-02

“For example, DeepSeek-V3 [1], still a relatively large model:”

举 DeepSeek-V3 为例,说明较大模型预训练用了 14.8 万亿 token
WD and Seagate confirm: Hard drives sold out for 2026原评论 ↗
引用jychang · 2026-03

“That's deepseek using FP8 live weight copy + FP32 master + FP32 grad + BF16 moments = 13 bytes per parameter.”

说明 DeepSeek 的 FP8 训练每个参数占用约 13 字节
No, it doesn't cost Anthropic $5k per Claude Code user原评论 ↗
引用fulafel · 2026-05

“DeepSeek V3 shipped MTP in a product first, in 2024:”

指出 DeepSeek V3 在 2024 年就率先在产品中落地 MTP
Accelerating Gemma 4: faster inference with multi-token prediction drafters原评论 ↗