“The article references Dixit et al. for an example of a root cause investigation of a CEE which is an interesting read”
Silent Data Corruptions at Scale
Meta 数据中心 CPU 静默数据损坏的缺陷类型、调试流程与缓解
静默数据损坏(SDC)不会被 CPU 内的错误报告机制捕捉,因此在硬件层面无法追踪,但会向上传播并表现为应用层问题,可能导致数据丢失并耗费数月调试时间。论文描述了导致 SDC 的常见硅制造缺陷类型,讨论数据中心应用中的真实案例,用案例演示如何定位出错的 CPU 指令,并概述降低风险的缓解措施。作者在数十万台机器上运行大量静默错误测试场景,检测到数百颗存在此类错误的 CPU,监测超过 18 个月,认为降低 SDC 不仅需要硬件韧性和生产检测,还需要健壮的容错软件架构。
HN 上的人把它当作证明静默数据损坏在大规模下真实存在的证据,认为可怕,用来说明位翻转和 CPU 悄悄算错在服务器上并不罕见。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Read the paper below. Absolutely frightening, we all basically swimming in quicksand.”
“Bit flips happen in networking, storage, buses between everything, caches, CPUs, etc. See eg [1]”
“Given Intel's AVX extension could cause silent failures on servers (very high work load for prolonged time, compare to end user computers), I'm not sure it would be a big win for servers either: Silent Data Corruptions at Scale.”
“there is a non zero number of servers giving unexpected results at scale”
- Silent Data Corruptions at Scale ↗jsty · 2021-03-03 · 1 分
- Silent Data Corruptions at Scale ↗fulafel · 2021-06-26 · 2 分
- Silent Data Corruptions at Scale ↗fulafel · 2021-07-04 · 2 分
- Silent data corruptions at scale (2021) ↗losfair · 2024-02-18 · 84 分
- Silent Data Corruptions at Scale ↗rbanffy · 2026-06-29 · 1 分