“Have a look at[1]”
GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers
基于二阶信息的一次性权重量化,把大模型压到 3–4 比特
论文提出 GPTQ,一种基于近似二阶信息的一次性权重量化方法,约 4 个 GPU 小时即可量化 1750 亿参数的 GPT 模型,把每个权重降到 3 或 4 比特,相对未压缩基线的精度损失可忽略。它首次让 1750 亿参数模型能在单块 GPU 上做生成推理,端到端推理在 A100 上约快 3.25 倍、在 A6000 上约快 4.5 倍;在 2 比特甚至三值量化的极端设置下也仍有可接受的精度。
评论者在讨论本地运行 LLaMA 等模型时引用它,说明 4 比特甚至 3 比特量化几乎不损失质量、能省下约 75% 显存,也有人指出它的主要基准是 WikiText2 困惑度,对多数用户意义有限。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“If you'd like to know more, I think you should read GPTQ paper[2].”
“Check out the original paper on quantization, which has some benchmarks: GPTQ”
“Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users.”
“text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1]”
“The quantization method they use is from the GPTQ paper which leads to almost no quality degradation compared to the 16bit weights.”
看全部 17 条 →
“However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead.”
“If they were using techniques like GPTQ, presumably it would work much better.”
“Decrease in accuracy is negligible and decreases as model size increases.”
“WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization.”
“GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16.”
“They're using GTPQ -- here you go: GPTQ. The authors benchmarked two families of models over a wide range of numbers of params.”
“As far as I know, yes. GPTQ”
“GPTQ (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy.”
“according to the GPTQ paper(§) the quality of the model will not suffer much at all by the quantization.”
“I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it.”
“one evidence is in the GPTQ paper”
- GPTQ Accurate Post-Training Quantization for Generative Pre-Trained Transformers ↗tosh · 2023-05-01 · 2 分