按年被提到
20242026 · 2 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2402.18158 · cs.CL

Evaluating Quantized Large Language Models

Shiyao Li, Xuefei Ning, Luning Wang, Tengxuan Liu, Xiangsheng Shi, Shengen Yan, Guohao Dai, Huazhong Yang, Yu Wang · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-03-01 · 最近 2026-07-14
2不同的人
2活跃月份
2讨论串
53被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Post-training quantization (PTQ) has emerged as a promising technique to reduce the cost of large language models (LLMs). Specifically, PTQ can effectively mitigate memory consumption and reduce computational overhead in LLMs. To meet the requirements of both high efficiency and performance across diverse scenarios, a comprehensive evaluation of quantized LLMs is essential to guide the selection of quantization methods. This paper presents a thorough evaluation of these factors by evaluating the effect of PTQ on Weight, Activation, and KV Cache on 11 model families, including OPT, LLaMA2, Falcon, Bloomz, Mistral, ChatGLM, Vicuna, LongChat, StableLM, Gemma, and Mamba, with parameters ranging from 125M to 180B. The evaluation encompasses five types of tasks: basic NLP, emergent ability, trustworthiness, dialogue, and long-context tasks. Moreover, we also evaluate the state-of-the-art (SOTA) quantization methods to demonstrate their applicability. Based on the extensive experiments, we systematically summarize the effect of quantization, provide recommendations to apply quantization techniques, and point out future directions. The code can be found in https://github.com/thu-nics/qllm-eval.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论verdverm · 2026-07

“Preliminary analysis via lm-evaluation-harness + vllm model | disk | wikitext | gsm8k (match/error) baseline | 55G | 8.00 | 0.50/0.09 nvfp4-gptq | 27G | 8.25 | 0.47/0.9 nvfp4a16-gptq | 27G | 8.11 | 0.53/0.9 bonsai-4bit | 19G | 16.75 | 0/0 Looks like they quant'd too hard at 4 bits, can't imagine the ternary being any good based on this. Code if you…”

Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone原评论 ↗
评论verdverm · 2026-07

“4bits is a cutoff point for many model families, but also depends on what parts you quant to 4bits vs alternatives (weights, weight+activation, kv cache). Also depends on model size and task, lots of nuance in quanting I've come to learn. Good evaluation from 2024 https://arxiv.org/pdf/2402.18158 I'm currently working towards an updated version (not an …”

Bonsai 27B (1-bit LLM): The First 27B-Class Model to Run on a Phone原评论 ↗
被提交为 story · 1 次