按年被提到
20232026 · 2 次2026
机器学习与深度学习›训练加速与低精度计算·arXiv:2310.10537 · cs.LG

Microscaling Data Formats for Deep Learning

Bita Darvish Rouhani, Ritchie Zhao, Ankit More, Mathew Hall, Alireza Khodamoradi, Summer Deng, Dhruv Choudhary, Marius Cornea, Eric Dellinger, Kristof Denolf, Stosic Dusan, Venmugil Elango 等 33 人 · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-10-18 · 最近 2026-04-07
2不同的人
4活跃月份
4讨论串
60被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Narrow bit-width data formats are key to reducing the computational and storage costs of modern deep learning applications. This paper evaluates Microscaling (MX) data formats that combine a per-block scaling factor with narrow floating-point and integer types for individual elements. MX formats balance the competing needs of hardware efficiency, model accuracy, and user friction. Empirical results on over two dozen benchmarks demonstrate practicality of MX data formats as a drop-in replacement for baseline FP32 for AI inference and training with low user friction. We also show the first instance of training generative language models at sub-8-bit weights, activations, and gradients with minimal accuracy loss and no modifications to the training recipe.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论opcode84 · 2023-10

“Earlier this year, AMD, Arm, Intel, Meta, Microsoft, NVIDIA, and Qualcomm Technologies, Inc. formed the Microscaling Formats (MX) Alliance with the goal of creating and standardizing next-generation 6- and 4-bit data types for AI training and inferencing. The key enabling technology that enables sub 8-bit formats to work, referred to as microscaling, buil…”

AMD, Arm, Intel, Meta, Microsoft, Nvidia, Qualcomm Standardize New AI Datatypes原评论 ↗
评论buildbot · 2024-03

“FP19 is perfectly fine for training - llama is trained in fp16, many train in bf16, with microscaling exponents, you can do 6 bit training: https://arxiv.org/abs/2310.10537”

Tenstorrent unveils Grayskull, its RISC-V answer to GPUs原评论 ↗
评论buildbot · 2026-03

“You can even train in 4 & 8 bits with newer microscaled formats! From https://arxiv.org/pdf/2310.10537 to gpt-oss being trained (partially) natively in MXFP4 - https://huggingface.co/blog/RakshitAralimatti/learn-ai-with-me To Nemotron 3 Super, which had 25T of nvfp4 native pretraining! https://docs.nvidia.com/nemotron/0.1.0/nemotron/super3/pretrain.html”

Quantization from the Ground Up原评论 ↗
评论buildbot · 2026-04

“Super cool to tape out a nearly 500MHz systolic array!!! I wish sometimes I had gone far enough with my hardware education to be able to do more than simple FOGA designs. A really cool thing to see would be the newer block scaled fp4/fp8 data types. Maybe for their next asic they can try it haha - https://arxiv.org/abs/2310.10537”

Floating point from scratch: Hard Mode原评论 ↗