“1-bit seems to be a very wrong name for {-1, 0, 1}.”
The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits
权重取 {-1,0,1} 的三值 LLM,号称与 FP16 持平且更省
论文提出 BitNet b1.58,一种 1-bit LLM 变体,其每个参数都是三值 {-1, 0, 1}。作者称在相同模型规模和训练 token 数下,它在困惑度和下游任务上与全精度(FP16 或 BF16)Transformer 相当,同时在延迟、内存、吞吐和能耗上明显更划算。论文还主张这给出了训练高性能且低成本新一代 LLM 的缩放定律与配方,并带来新的计算范式,为专用于 1-bit LLM 的硬件设计打开了空间。
评论者围绕三值权重(-1/0/1)能否用整数加法取代浮点乘加、能否在 FPGA 或专用硬件上跑得更快来讨论,也有人认为“1-bit”命名不准,并怀疑实际可用性不如论文所示。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“I get the impression that there are factors not considered in the paper which make it less practical than it seemed.”
“On the contrary, in the BitNet b1.56 paper [0] the authors report their 7b model has 2.9x reduced inference latency.”
“Those proposed ternary LLMs could potentially run much faster on FPGAs, because the hardware could be optimized for exclusively ternary ops.”
“Especially with the 2-bit data types outlined in that paper [0] and more details in [1].”
“This is a more recent paper by the same people, where they tweak slightly to ternary (-1, 0, 1), and they seem to be able to maintain network performance:”
看全部 17 条 →
“where you replace the nn.Linear layer with a BitLinear layers as introduced in the paper on 1.58 bit LLMs The Era of 1-bit LLMs”
“The quantization approach is basically identical to the 1.58bit LLM paper:”
“this is a real prototype for Bitnet 1.58b which uses (-1,0,1) as the weights”
“They say 1.58 in other places and explicitly link to The Era of 1-bit LLMs.”
“It matches the full-precision (i.e., FP16 or BF16) Transformer LLM with the same model size and training tokens in terms of both perplexity and end-task performance”
“This seems to be first source of the confounding of”
“Most of the weights in Bitnet b1.58[1] class models[2][3] are ternary (-1/0/1).”
“Also, could this The Era of 1-bit LLMs possibly reduce errors when doing inference? There is no floating point operations”
“Didn't this paper demonstrate that you only need 1.58 bits to be equivalent to 16 bits in performance?”
“since the original paper (Figure 3) shows that we eliminate FP16 multiplication and addition just for INT8 addition.”
“And it looks like per [0], a model needs only ~30% more weights to be at comparable quality, if quantization-aware training is done...”
- The Era of 1-bit LLMs: ternary parameters for cost-effective computing ↗fgfm · 2024-02-28 · 1040 分
- The Era of 1-Bit LLMs: All Large Language Models Are in 1.58 Bits ↗Luc · 2024-02-28 · 2 分
- 1.58-bit LLMs: thousands of tokens/sec, reduced computing footprint ↗Rabot · 2024-03-01 · 1 分