“Paper: https://arxiv.org/abs/2406.02528 -- always better than a press release. Code: https://github.com/ridgerchu/matmulfreellm --- Like others before them, the authors train LLMs using parameters consisting of ternary digits, or trits, with values in {-1, 0, 1}. What's new is that the authors then build a custom hardware solution on an FPGA and run b…”
Scalable MatMul-free Language Modeling
Large Language Models (LLMs) have fundamentally altered how we approach scaling in machine learning. However, these models pose substantial computational and memory challenges, primarily due to the reliance on matrix multiplication (MatMul) within their attention and feed-forward (FFN) layers. We demonstrate that MatMul operations can be eliminated from LLMs while maintaining strong performance, even at billion-parameter scales. Our MatMul-free models, tested on models up to 2.7B parameters, are comparable to state-of-the-art pre-trained Transformers, and the performance gap narrows as model size increases. Our approach yields significant memory savings: a GPU-efficient implementation reduces memory consumption by up to 61% during training and over 10x during inference. When adapted for a multi-chip neuromorphic system, the model leverages asynchronous processing to achieve 4x higher throughput with 10x less energy than edge GPUs.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“The press release is devoid of useful information, unsurprisingly. You can run an LLM under almost any energy envelope if you’re willing to wait long enough for the result. Total energy consumed and the time difference are the more important metrics. The actual paper is here: https://arxiv.org/abs/2406.02528 The key part from the summary: > To properly…”
“The relevant paper: https://arxiv.org/abs/2406.02528 In summary, they forced the model to process data in ternary system and then build a custom FPGA chip to process the data more efficiently. Tested to be "comparable" to small models (3B), theoretically scale to 70B, unknown for SOTAs (>100B params). We have always known custom chips are more efficient…”
“Fantastic result, on par with another similar effort: https://arxiv.org/pdf/2406.02528 It seems to me that we've stumbled upon this method of GPU-heavy matrix-multiplications in deep neural nets, and have only scratched the surface of alternative methods that are actually optimized for current CPU architectures such as Tsetlin Machines, Hyperdimensional …”
- Scalable MatMul-Free Language Modeling ↗optimalsolver · 2024-06-07 · 3 分
- Scalable MatMul-Free Language Modeling ↗throwaway71271 · 2024-06-08 · 3 分
- Scalable MatMul-Free Language Modeling ↗lykahb · 2024-06-09 · 205 分