“Turns out models trained in BF16 are much more quantizable to INT8 and INT4 over their FP16 counterparts but the paper doesn't delve much into the reason for this. Anyone has insights as to why this might be happening?”
Intriguing Properties of Quantization at Scale
Emergent properties have been widely adopted as a term to describe behavior not present in smaller models but observed in larger models. Recent work suggests that the trade-off incurred by quantization is also an emergent property, with sharp drops in performance in models over 6B parameters. In this work, we ask "are quantization cliffs in performance solely a factor of scale?" Against a backdrop of increased research focus on why certain emergent properties surface at scale, this work provides a useful counter-example. We posit that it is possible to optimize for a quantization friendly training recipe that suppresses large activation magnitude outliers. Here, we find that outlier dimensions are not an inherent product of scale, but rather sensitive to the optimization conditions present during pre-training. This both opens up directions for more efficient quantization, and poses the question of whether other emergent properties are inherent or can be altered and conditioned by optimization and architecture design choices. We successfully quantize models ranging in size from 410M to 52B with minimal degradation in performance.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)- Intriguing Properties of Quantization at Scale ↗Jimmc414 · 2023-05-31 · 2 分
- Train your Large Language Models in BF16 over FP16 ↗bigmodelenergy · 2023-07-02 · 1 分