“This it the best explanation I've read of how memory is allocated to training deep learning models, and what possible solutions there are to reducing that footprint. Some things I already knew about, such as gradient checkpointing and FP16. What was new to me is microbatching (gradient accumulation). Many of the large models appearing these days (Transf…”
Low-Memory Neural Network Training: A Technical Report
Memory is increasingly often the bottleneck when training neural network models. Despite this, techniques to lower the overall memory requirements of training have been less widely studied compared to the extensive literature on reducing the memory requirements of inference. In this paper we study a fundamental question: How much memory is actually needed to train a neural network? To answer this question, we profile the overall memory usage of training on two representative deep learning benchmarks -- the WideResNet model for image classification and the DynamicConv Transformer model for machine translation -- and comprehensively evaluate four standard techniques for reducing the training memory requirements: (1) imposing sparsity on the model, (2) using low precision, (3) microbatching, and (4) gradient checkpointing. We explore how each of these techniques in isolation affects both the peak memory usage of training and the quality of the end model, and explore the memory, accuracy, and computation tradeoffs incurred when combining these techniques. Using appropriate combinations of these techniques, we show that it is possible to the reduce the memory required to train a WideResNet-28-2 on CIFAR-10 by up to 60.7x with a 0.4% loss in accuracy, and reduce the memory required to train a DynamicConv model on IWSLT'14 German to English translation by up to 8.7x with a BLEU score drop of 0.15.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)“I feel like Tensorflow tends to allocate scratch space in GPU RAM for the result of every math operation. So if you have a big tensor and want to do math on it, you need space to effectively store multiple copies of the tensor. I think TF does this for speed reasons.”
- Low-Memory Neural Network Training: A Technical Report ↗nimz12 · 2019-04-25 · 8 分
- Low-Memory Neural Network Training: A Technical Report ↗wildermuthn · 2019-09-21 · 58 分