按年被提到
20232023 · 2 次2026
机器学习与深度学习›贝叶斯方法与 AutoML·arXiv:2305.19268 · cs.LG

Intriguing Properties of Quantization at Scale

Arash Ahmadian, Saurabh Dash, Hongyu Chen, Bharat Venkitesh, Stephen Gou, Phil Blunsom, Ahmet Üstün, Sara Hooker · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-05-31 · 最近 2023-07-02
2不同的人
2活跃月份
2讨论串
13被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models. Recent work suggests that the trade-off incurred by quantization is also an emergent property, with sharp drops in performance in models over 6B parameters. In this work, we ask "are quantization cliffs in performance solely a factor of scale?" Against a backdrop of increased research focus on why certain emergent properties surface at scale, this work provides a useful counter-example. We posit that it is possible to optimize for a quantization friendly training recipe that suppresses large activation magnitude outliers. Here, we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training. This both opens up directions for more efficient quantization, and poses the question of whether other emergent properties are inherent or can be altered and conditioned by optimization and architecture design choices. We successfully quantize models ranging in size from 410M to 52B with minimal degradation in performance.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论bigmodelenergy · 2023-07

“Turns out models trained in BF16 are much more quantizable to INT8 and INT4 over their FP16 counterparts but the paper doesn't delve much into the reason for this. Anyone has insights as to why this might be happening?”

Train your Large Language Models in BF16 over FP16原评论 ↗
被提交为 story · 2 次