推荐dwd · 2023-12
“The paper is worth a read in full as what they are doing is pretty cool:”
推荐通读这篇论文,它用闪存加载技巧运行超出 DRAM 的 LLM,做法很酷
把模型参数放闪存、按需载入 DRAM,让大模型跑在小内存设备上
针对模型大于 DRAM 的情形,把参数存在闪存里按需调入内存。作者先建立考虑闪存特性的推理成本模型,再提出两项技术:利用已激活神经元的 windowing 以减少数据传输,以及 row-column bundling 以增大顺序读取块。这使得可运行的模型最大达到可用 DRAM 的两倍,推理速度相比朴素加载在 CPU 和 GPU 上分别提升 4–5 倍和 20–25 倍。
HN 上的人把它当作苹果在端侧运行大模型的代表研究,用来解释为什么快速 SSD 对本地跑大模型很重要。
“The paper is worth a read in full as what they are doing is pretty cool:”
“Apple’s “LLM in a flash” paper [1] sheds some light on why this can be a big deal”
“Also relevant: LLM in a flash: Efficient Large Language Model Inference with Limited Memory”
“This indeed seems to have been a heavy focus of their research team in the past year”
“It's apparently using ideas from: LLM in a flash”