“so check out the paper if you're interested.”
BitNet: Scaling 1-bit Transformers for Large Language Models
用 BitLinear 从头训练 1 比特权重的 Transformer
论文提出 BitNet,一种可扩展且稳定的 1 比特 Transformer 架构,用 BitLinear 作为 nn.Linear 的替换件,从头训练 1 比特权重。语言建模实验表明,它在显著降低内存占用和能耗的同时,性能与最先进的 8 比特量化方法和 FP16 Transformer 基线相当,并呈现出与全精度 Transformer 类似的缩放规律。
评论者用它说明低比特权重(原版为 ±1 二值,后续为三值)的 LLM 是可行的、适合受限硬件,并查证输入量化、参数需求等细节,也有人批评其训练成本改善有限。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The only paper they cite for training: BitNet doesn't improve training costs much”
“The other paper had a much nicer and clearer introduction to bitlinear than the original Microsoft paper, which is why I used it.”
“It seems to have more details (it's the paper before the linked one) about the actual training”
“The details are in the original bitnet paper: BitNet”
“BitNet: Scaling 1-bit Transformers for Large Language Models was actually binary (weights of -1 or 1),”
看全部 10 条 →
“I wonder how well BitNet (ternary weights) would work for this. It seems like a promising way forward for constrained hardware.”
“But the fact that BitNet [2] is possible calls into question the need for it.”
“The original paper [fig 1, bottom-right] seems to say it needs about 4-5x the parameters of a fp16 model.”
“since the bitnet paper has been out for a couple years now: BitNet”
- BitNet: Scaling 1-Bit Transformers for Large Language Models ↗kevin_hu · 2023-10-18 · 3 分