“Also greedy sampling considered harmful: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference”
Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference
GPU数目、批大小等配置会因浮点舍入让LLM输出不一致
论文指出大模型成绩的可复现性很脆弱:评测批大小、GPU 数量和 GPU 版本等系统配置变化,就会让生成结果出现显著差异,推理模型尤其明显,因为早期 token 的微小舍入差异会级联成不同的思维链。例如在 bfloat16 与贪心解码下,DeepSeek-R1-Distill-Qwen-7B 的准确率最多波动 9%,响应长度相差约 9000 个 token。作者把根源追溯到有限精度下浮点运算不满足结合律,并提出轻量推理流程 LayerCast:以 16 位存权重、全部计算用 FP32,在显存效率与数值稳定间折中。
HN 评论者引它反驳“把温度设为 0 就能得到确定性输出”的误解,说明浮点运算和 GPU 非确定性会让 LLM 推理结果不可复现。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“See Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference for some details on that.”
“I wanted to point out this paper: Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inference which analyzes the issues around GPU non determinism”
“only repeating a known misconception of tweaking the temperature to 0 which does not bring the determinism you claim it brings with LLMs [0] [1] [2]”
“Floating point matrix calculations are non-deterministic.”
- Give Me FP32 or Give Me Death? ↗spindump8930 · 2025-06-25 · 3 分
- Understanding, Mitigating Numerical Sources of Nondeterminism in LLM Inference ↗shakna · 2026-09-03 · 1 分