按年被提到
20242024 · 12 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2402.17764 · cs.CL

The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits

1-bit LLM 时代:所有大语言模型都只需 1.58 比特
Shuming Ma, Hongyu Wang, Lingxiao Ma, Lei Wang, Wenhui Wang, Shaohan Huang, Li Dong, Ruiping Wang, Jilong Xue, Furu Wei · 2024

权重取 {-1,0,1} 的三值 LLM,号称与 FP16 持平且更省

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 21 人首次 2024-02-28 · 最近 2026-09-17
21不同的人
12活跃月份
19讨论串
167被引(OpenAlex)

论文提出 BitNet b1.58,一种 1-bit LLM 变体,其每个参数都是三值 {-1, 0, 1}。作者称在相同模型规模和训练 token 数下,它在困惑度和下游任务上与全精度(FP16 或 BF16)Transformer 相当,同时在延迟、内存、吞吐和能耗上明显更划算。论文还主张这给出了训练高性能且低成本新一代 LLM 的缩放定律与配方,并带来新的计算范式,为专用于 1-bit LLM 的硬件设计打开了空间。

HN 为什么提它

评论者围绕三值权重(-1/0/1)能否用整数加法取代浮点乘加、能否在 FPGA 或专用硬件上跑得更快来讨论,也有人认为“1-bit”命名不准,并怀疑实际可用性不如论文所示。

方法难度 3 / 5约 20 分钟经得起时间
读之前最好知道
1Transformer 语言模型
2模型量化(quantization)与量化感知训练
3困惑度(perplexity)
4缩放定律(scaling laws)

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
批评bmacho · 2024-02

“1-bit seems to be a very wrong name for {-1, 0, 1}.”

The Era of 1-bit LLMs: ternary parameters for cost-effective computing原评论 ↗
批评lioeters · 2026-07

“I get the impression that there are factors not considered in the paper which make it less practical than it seemed.”

How to pack ternary numbers in 8-bit bytes原评论 ↗
引用jstmm · 2024-03

“On the contrary, in the BitNet b1.56 paper [0] the authors report their 7b model has 2.9x reduced inference latency.”

反驳时引用论文:7B 模型推理延迟降低 2.9 倍
Towards 1-bit Machine Learning Models原评论 ↗
引用zachbee · 2024-04

“Those proposed ternary LLMs could potentially run much faster on FPGAs, because the hardware could be optimized for exclusively ternary ops.”

指出论文提出的三值 LLM 在 FPGA 上可能跑得更快
FPGA Architecture for Deep Learning: Survey and Future Directions原评论 ↗
引用UncleOxidant · 2024-05

“Especially with the 2-bit data types outlined in that paper [0] and more details in [1].”

引用论文提出的 2-bit 数据类型,说明为 LLM 计算定制硬件的动机
GPUs Go Brrr原评论 ↗
引用ericd · 2024-06

“This is a more recent paper by the same people, where they tweak slightly to ternary (-1, 0, 1), and they seem to be able to maintain network performance:”

介绍同一批人的新论文:改用三值(-1、0、1)权重,据称性能不降
AMD Unveils Ryzen 9000 CPUs for Desktop, Zen 5原评论 ↗
看全部 17 条 →
引用cafed00d · 2024-06

“where you replace the nn.Linear layer with a BitLinear layers as introduced in the paper on 1.58 bit LLMs The Era of 1-bit LLMs”

说明在小型 GPT 中用论文提出的 BitLinear 层替换 nn.Linear
Show HN: A Small BitNet GPT in MLX原评论 ↗
引用cpldcpu · 2024-06

“The quantization approach is basically identical to the 1.58bit LLM paper:”

指出量化方法与 1.58 bit LLM 论文基本相同
Scalable MatMul-Free Language Modeling原评论 ↗
引用syntaxing · 2024-06

“this is a real prototype for Bitnet 1.58b which uses (-1,0,1) as the weights”

指出这是 BitNet 1.58b 的真实原型,权重取 (-1,0,1)
Researchers run high-performing LLM on the energy needed to power a lightbulb原评论 ↗
引用cubefox · 2025-01

“They say 1.58 in other places and explicitly link to The Era of 1-bit LLMs.”

指出论文别处就写 1.58 并明确链接到这篇,是 1.58 比特叫法的来源
Run DeepSeek R1 Dynamic 1.58-bit原评论 ↗
引用hu3 · 2025-04

“It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance”

引用论文声称:同尺寸同训练 token 下困惑度和任务表现与全精度持平
Microsoft researchers developed a hyper-efficient AI model that can run on CPUs原评论 ↗
引用AzN1337c0d3r · 2025-04

“This seems to be first source of the confounding of”

指出这是“1-bit”命名混淆的最初来源,把 1-bit LLM 与三值权重混为一谈
Microsoft researchers developed a hyper-efficient AI model that can run on CPUs原评论 ↗
引用woadwarrior01 · 2025-10

“Most of the weights in Bitnet b1.58[1] class models[2][3] are ternary (-1/0/1).”

说明 BitNet b1.58 类模型的权重大多是三值 (-1/0/1)
The Trinary Dream Endures原评论 ↗
引用kovek · 2025-10

“Also, could this The Era of 1-bit LLMs possibly reduce errors when doing inference? There is no floating point operations”

想问它没有浮点运算,推理时是否可能减少误差
Why can't transformers learn multiplication?原评论 ↗
引用codexon · 2026-01

“Didn't this paper demonstrate that you only need 1.58 bits to be equivalent to 16 bits in performance?”

回忆这篇论文是否证明了只需 1.58 比特就能达到 16 比特的表现
Kimi Released Kimi K2.5, Open-Source Visual SOTA-Agentic Model原评论 ↗
引用c0rruptbytes · 2026-05

“since the original paper (Figure 3) shows that we eliminate FP16 multiplication and addition just for INT8 addition.”

引用原论文图 3:只需 INT8 加法,省掉 FP16 乘加
1-Bit Bonsai Image 4B Image Generation for Local Devices原评论 ↗
引用yalok · 2026-09

“And it looks like per [0], a model needs only ~30% more weights to be at comparable quality, if quantization-aware training is done...”

引用它说明做量化感知训练时,模型约多 30% 权重即可达到相当质量
Breaking the 1.58-bit Barrier for Ternary LLMs原评论 ↗
被提交为 story · 3 次