按年被提到
20222023 · 1 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2205.13016 · cs.LG

BiT: Robustly Binarized Multi-distilled Transformer

Zechun Liu, Barlas Oguz, Aasish Pappu, Lin Xiao, Scott Yih, Meng Li, Raghuraman Krishnamoorthi, Yashar Mehdad · 2022
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-09-06 · 最近 2025-01-02
2不同的人
2活跃月份
2讨论串
43被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however, is technically challenging from an optimization perspective. In this work, we identify a series of improvements that enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, a novel elastic binary activation function with learned parameters, and a method to quantize a network to its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%. Code and models are available at: https://github.com/facebookresearch/bit.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论astrange · 2025-01

“Treating it as continuous is a property of the training algorithm, but there are networks that use binary values. https://ieeexplore.ieee.org/document/9359148 https://arxiv.org/abs/2205.13016”

Coconut by Meta AI – Better LLM Reasoning with Chain of Continuous Thought?原评论 ↗
被提交为 story · 1 次