“I am curious if there is anyone familiar with the standardization progress of those 8-bit representations? PS: Assuming they can only have 256 unique values, would have been great if the papers provided conversion tables for every 8-bit float to a 32-bit value, to make testing accuracy degradation easier across applications.”
8-bit Numerical Formats for Deep Neural Networks
Given the current trend of increasing size and complexity of machine learning architectures, it has become of critical importance to identify new approaches to improve the computational efficiency of model training. In this context, we address the advantages of floating-point over fixed-point representation, and present an in-depth study on the use of 8-bit floating-point number formats for activations, weights, and gradients for both training and inference. We explore the effect of different bit-widths for exponents and significands and different exponent biases. The experimental results demonstrate that a suitable choice of these low-precision formats enables faster training and reduced power consumption without any degradation in accuracy for a range of deep learning models for image classification and language processing.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)- 8-Bit Numerical Formats for Deep Neural Networks ↗ingve · 2022-06-08 · 2 分
- 8-Bit Numerical Formats for Deep Neural Networks (2022) ↗ashvardanian · 2023-07-18 · 1 分