按年被提到
20232024 · 5 次2026
大模型与语言模型›量化、解码与分词·arXiv:2310.11453 · cs.CL

BitNet: Scaling 1-bit Transformers for Large Language Models

BitNet:为大语言模型扩展 1 比特 Transformer
Hongyu Wang, Shuming Ma, Li Dong, Shaohan Huang, Huaijie Wang, Lingxiao Ma, Fan Yang, Ruiping Wang, Yi Wu, Furu Wei · 2023

用 BitLinear 从头训练 1 比特权重的 Transformer

开始阅读 ↗PDFalphaXiv 讨论
被推荐 · 10 人首次 2023-10-18 · 最近 2026-07-28
10不同的人
9活跃月份
9讨论串
108被引(OpenAlex)

论文提出 BitNet,一种可扩展且稳定的 1 比特 Transformer 架构,用 BitLinear 作为 nn.Linear 的替换件,从头训练 1 比特权重。语言建模实验表明,它在显著降低内存占用和能耗的同时,性能与最先进的 8 比特量化方法和 FP16 Transformer 基线相当,并呈现出与全精度 Transformer 类似的缩放规律。

HN 为什么提它

评论者用它说明低比特权重(原版为 ±1 二值,后续为三值)的 LLM 是可行的、适合受限硬件,并查证输入量化、参数需求等细节,也有人批评其训练成本改善有限。

方法难度 4 / 5约 30 分钟经得起时间
读之前最好知道
1Transformer 架构
2神经网络量化(quantization)
3缩放定律(scaling law)
4混合精度训练

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
推荐Centigonal · 2024-03

“so check out the paper if you're interested.”

推荐感兴趣的人读这篇论文,它从零训练低比特 LLM
Towards 1-bit Machine Learning Models原评论 ↗
批评llm_trw · 2024-02

“The only paper they cite for training: BitNet doesn't improve training costs much”

The Era of 1-bit LLMs: ternary parameters for cost-effective computing原评论 ↗
批评llm_trw · 2024-11

“The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it.”

SVDQuant: 4-Bit Quantization Powers 12B Flux on a 16GB 4090 GPU with 3x Speedup原评论 ↗
引用IanCal · 2024-02

“It seems to have more details (it's the paper before the linked one) about the actual training”

指出这是链接那篇之前的论文,有更多 1 比特 Transformer 的训练细节
The Era of 1-bit LLMs: ternary parameters for cost-effective computing原评论 ↗
引用elromulous · 2024-02

“The details are in the original bitnet paper: BitNet”

指出细节在最初的 BitNet 论文里:输入被量化,运算用整数完成
The Era of 1-bit LLMs: ternary parameters for cost-effective computing原评论 ↗
引用AzN1337c0d3r · 2025-04

“BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1),”

指出最初的 BitNet 其实是二值的,权重为 -1 或 1
Microsoft researchers developed a hyper-efficient AI model that can run on CPUs原评论 ↗
看全部 10 条 →
引用mattdesl · 2025-05

“I wonder how well BitNet (ternary weights) would work for this. It seems like a promising way forward for constrained hardware.”

好奇 BitNet 的三值权重用在这里效果如何,认为是受限硬件的有前景方向
Towards the Cutest Neural Network原评论 ↗
引用fc417fc802 · 2025-06

“But the fact that BitNet [2] is possible calls into question the need for it.”

拿 BitNet 的可行性质疑对数运算是否有必要
Posit floating point numbers: thin triangles and other tricks (2019)原评论 ↗
引用herf · 2026-03

“The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model.”

引用原论文图 1,说 1 比特模型需要 fp16 模型约 4 到 5 倍的参数
Microsoft BitNet: 100B Param 1-Bit model for local CPUs原评论 ↗
引用kamranjon · 2026-07

“since the bitnet paper has been out for a couple years now: BitNet”

指出 BitNet 论文已出两年,原生 1.58 比特训练不同于训练后量化
Neutrino-1 8B原评论 ↗
被提交为 story · 1 次