按年被提到
20222023 · 18 次2026
大模型与语言模型›量化、解码与分词·arXiv:2210.17323 · cs.LG

GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers

GPTQ:面向生成式预训练 Transformer 的精确训练后量化
Elias Frantar, Saleh Ashkboos, Torsten Hoefler, Dan Alistarh · 2022

基于二阶信息的一次性权重量化,把大模型压到 3–4 比特

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 15 人首次 2023-03-10 · 最近 2023-12-16
15不同的人
4活跃月份
13讨论串
727被引(OpenAlex)

论文提出 GPTQ,一种基于近似二阶信息的一次性权重量化方法,约 4 个 GPU 小时即可量化 1750 亿参数的 GPT 模型,把每个权重降到 3 或 4 比特,相对未压缩基线的精度损失可忽略。它首次让 1750 亿参数模型能在单块 GPU 上做生成推理,端到端推理在 A100 上约快 3.25 倍、在 A6000 上约快 4.5 倍;在 2 比特甚至三值量化的极端设置下也仍有可接受的精度。

HN 为什么提它

评论者在讨论本地运行 LLaMA 等模型时引用它,说明 4 比特甚至 3 比特量化几乎不损失质量、能省下约 75% 显存,也有人指出它的主要基准是 WikiText2 困惑度,对多数用户意义有限。

方法难度 4 / 5约 50 分钟经得起时间
读之前最好知道
1神经网络量化基础
2Transformer 与 GPT 架构
3二阶优化与 Hessian
4困惑度(perplexity)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐garblegarble · 2023-03

“Have a look at[1]”

推荐看它,一种 3-4 位量化方法,可缩小 LLM 内存需求
Apple: Transformer architecture optimized for Apple Silicon原评论 ↗
推荐terafo · 2023-03

“If you'd like to know more, I think you should read GPTQ paper[2].”

推荐读 GPTQ 论文,想了解量化如何影响模型质量
Llama.cpp 30B runs with only 6GB of RAM now原评论 ↗
推荐corvec · 2023-03

“Check out the original paper on quantization, which has some benchmarks: GPTQ”

推荐看量化原始论文,里面有一些基准测试
Llama.cpp 30B runs with only 6GB of RAM now原评论 ↗
批评rnosov · 2023-03

“Their headline benchmark is perplexity on WikiText2 which would not be particularly relevant to most users.”

The genie escapes: Stanford copies the ChatGPT AI for less than $600原评论 ↗
引用MacsHeadroom · 2023-03

“text-generation-webui supports state of the art 4bit GPTQ quantization for LLaMA[0], reducing VRAM overhead by 75% with no output performance loss compared to baseline fp16.[1]”

引它说明 4 位 GPTQ 量化显存减少 75% 且输出表现无损
LLaMA Text-Generation-Webui原评论 ↗
引用underlines · 2023-03

“The quantization method they use is from the GPTQ paper which leads to almost no quality degradation compared to the 16bit weights.”

指出所用量化方法来自 GPTQ,与 16 位权重相比几乎没有质量下降
LLaMA-7B in Pure C++ with full Apple Silicon support原评论 ↗
看全部 17 条 →
引用MacsHeadroom · 2023-03

“However, even this minimal amount can be avoided with GPTQ quantization which maintains uncompressed fp16 performance even at 4bit quantization with 75% less (video)memory overhead.”

引它说明 GPTQ 量化即使 4 位也保持 fp16 表现,内存少 75%
LLaMA-7B in Pure C++ with full Apple Silicon support原评论 ↗
引用sebzim4500 · 2023-03

“If they were using techniques like GPTQ, presumably it would work much better.”

指出若用 GPTQ 这类技术,效果大概会好得多
Large language models are having their Stable Diffusion moment原评论 ↗
引用gamegoblin · 2023-03

“Decrease in accuracy is negligible and decreases as model size increases.”

引它说明量化的精度损失可忽略,且随模型变大而减小
Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up原评论 ↗
引用MacsHeadroom · 2023-03

“WRONG. Research shows effectively imperceptible performance difference at 4-bit and even 3-bit with GPTQ quantization.”

反驳对方,引研究说明 GPTQ 在 4 位甚至 3 位时表现差异几乎察觉不到
Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up原评论 ↗
引用underlines · 2023-03

“GPTQ doesn't quantize linearly. While RTN 8bit does reduce output quality, GPTQ 4bit has effectively little output quality loss compared to baseline uncompressed fp16.”

指出 GPTQ 不是线性量化,4 位输出质量与未压缩 fp16 相比几乎无损
Alpaca- An Instruct Tuned Llama 7B. Responses on par with txt-DaVinci-3. Demo up原评论 ↗
引用sottol · 2023-03

“They're using GTPQ -- here you go: GPTQ. The authors benchmarked two families of models over a wide range of numbers of params.”

指出对方用的是 GPTQ,并贴出论文,里面在大范围参数量上测了两个模型系列
LLaMa running at 5 tokens/second on a Pixel 6原评论 ↗
引用superkuh · 2023-03

“As far as I know, yes. GPTQ”

引它说明 175B 的 GPT 模型可量化到 3-4 位且精度损失可忽略
The genie escapes: Stanford copies the ChatGPT AI for less than $600原评论 ↗
引用sp332 · 2023-05

“GPTQ (GPTQ) showed that you can cut down the precision quite a bit without losing a lot of accuracy.”

引它说明可大幅降低精度而不损失太多准确率
How to run Llama 13B with a 6GB graphics card原评论 ↗
引用Tepix · 2023-05

“according to the GPTQ paper(§) the quality of the model will not suffer much at all by the quantization.”

引它说明量化对模型质量几乎没有影响
The tiny corp raised $5.1M原评论 ↗
引用mijoharas · 2023-09

“I had a quick look at the paper which is _reasonably_ clear, but if anyone else has any other sources that are easy to understand, or a quick explanation to give more insight into it, I'd appreciate it.”

说自己看过这篇论文,觉得还算清楚,想要更易懂的讲解
70B Llama 2 at 35tokens/second on 4090原评论 ↗
引用renonce · 2023-12

“one evidence is in the GPTQ paper”

指出 GPTQ 论文里有一条佐证
Do large language models need all those layers?原评论 ↗
被提交为 story · 1 次