“Read The Case for 4 Bit Precision. The case for 4-bit precision”
The case for 4-bit precision: k-bit Inference Scaling Laws
35,000 多组实验表明:模型总比特数固定时,4 比特量化几乎总是最优
同样的总比特数下,30B 的 8 比特模型和 60B 的 4 比特模型的零样本精度可能差别很大。作者建立 LLM 零样本性能的推理缩放定律,用 35,000 多次实验在 BLOOM、OPT、NeoX/Pythia 和 GPT-2 系列、19M 到 176B 参数、3 到 8 比特精度下比较各种量化方法。结果显示很难改善比特级缩放的权衡,仅有的改进来自较小的分块大小和量化数据类型(如 Int 对 Float),总体上 4 比特精度几乎普遍对总模型比特数和零样本精度最优。
HN 上的人反复用它说明 4 比特量化是模型大小与精度之间的最佳折中,认为几乎不损失输出质量,并在讨论本地运行大模型和降低推理成本时引用。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check out the original paper on quantization, which has some benchmarks: [OTHER_PAPER] and this paper, which also has benchmarks and explains how they determined that 4-bit quantization is optimal compared to 3-bit: The case for 4-bit precision”
“paper is a bit old, but matches current empirical recommandation”
“I think quantization (e.g. 4-bit, The case for 4-bit precision) and sparsity (e.g. SparseGPT, [OTHER_PAPER]) will bring down inference cost.”
“See also: The case for 4-bit precision, which shows effectively no output quality reduction for these 4bit quantization methods (and considerable speedup) The case for 4-bit precision”
“The case for 4-bit precision - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]”
看全部 14 条 →
“The case for 4-bit precision - The case for 4-bit precision: k-bit Inference Scaling Laws [Dec, 2022]”
“Tim Dettmers did a bunch of research on this last year as well: The case for 4-bit precision”
“The tradeoff evidently works favorably till upto 4-bits; I'm basing this off the results reported on the k-bit inference scaling laws paper”
“Generally 4bits+ achieves a good tradeoff, see eg. The case for 4-bit precision”
“Various papers have shown that 4-bit quantization is a great balance. One example: The case for 4-bit precision”
“(eg. The case for 4-bit precision )”
“arguably good results can be achieved even with 4 bit precision [0][1].”
“There are papers to that effect though admittedly perhaps specific models might have divergent behavior”
- The case for 4-bit precision: k-bit Inference Scaling Laws ↗Palmik · 2023-07-26 · 1 分