“You can read more on paper that my collegues in Google platforms recently published TPU v4”
TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
Google TPU v4 的光交换互连与稀疏核心设计
介绍 Google 第五代领域专用架构 TPU v4 及其第三代机器学习超级计算机。光路交换机(OCS)可动态重构互连拓扑,以提升规模、可用性、利用率、模块化、安全性、功耗和性能,用户还可选择扭曲三维环面拓扑;OCS 及光学部件占系统成本不到 5%、功耗不到 3%。每块芯片含 SparseCore,把依赖嵌入的模型加速 5 到 7 倍,仅占 5% 芯片面积和功耗。TPU v4 比 TPU v3 快 2.1 倍、每瓦性能高 2.7 倍,系统规模达 4096 芯片;论文称同规模下比 Nvidia A100 快 1.2 至 1.7 倍、功耗低 1.3 至 1.9 倍。
评论者在讨论 TPU 与 GPU 集群互连、光交换以及嵌入专用硬件时,把它当作 Google 官方公开的一手技术来源,称其内容引人入胜。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮你理解 TPU v4 面对的负载背景:Transformer 与大语言模型带来的算力需求,以及 AI 训练的能耗与碳排放问题。
- 1
- 2
GPT-3 代表了大规模模型对超大规模互连与并行训练的需求。
读读模型规模与训练算力的部分,体会为什么需要上千芯片的超级计算机。
- 3
- 4
TPU v4 的能效与碳排放数据在这里有更系统的论证。
读读四项最佳实践(模型、硬件、数据中心、地点)与硬件效率的论述。
- 5TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings
介绍光交换互连、可重构拓扑与稀疏核心的 TPU v4 超级计算机。
读重点读光路交换(OCS)如何带来可用性和扩展性,以及扭曲 3D 环面拓扑和嵌入的硬件支持。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The paper is pretty fascinating stuff.”
“Check out relatively recent Google TPU paper where they used an optical switch to connect neighboring TPUs”
“TPU's have a TPU-TPU interconnect that is faster and lower latency than any GPU cluster [1].”
“TPU v4 from April 4 of this year.”
“I can't find that chip's details anywhere beyond some mentions in the TPU v4 paper: TPU v4”
看全部 11 条 →
“so Google's TPUs have included a "sparse core" to separately handle those (the lookup and memory use patterns are drastically different from that of the typical dense matrix operations used for non-embedding layers) since TPUv2: TPU v4”
“Google claims the opposite in "TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings " TPU v4”
“Google publishes papers and gives talks about their TPUs.”
“sophisticated networking and switching topologies[1].”
“TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings”
- TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning ↗mfiguiere · 2023-04-05 · 1 分