“Here's an interesting follow up from a noted researcher (Tali Tishby) with one foot in the stat mech camp, and one in the information theory camp: Deep Learning and the Information Bottleneck Principle”
Deep Learning and the Information Bottleneck Principle
用信息瓶颈理论分析深度网络各层互信息与泛化界
用信息瓶颈(IB)原理分析深度神经网络:把网络量化为各层与输入、输出变量之间的互信息,据此计算信息论极限并得到有限样本泛化界。作者认为最优的层数和每层特征数与信息瓶颈权衡的分岔点有关,层级表示对应信息曲线上的结构性相变。
早期评论者把它当作连接统计力学、信息论与深度学习的有趣工作;后来有人称其为经典,但也有评论认为其关键主张与经验结果相矛盾。
学习路径
读这篇之前,按顺序读书架上的 1 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出候选中只有一篇与目标论文的理论视角直接相关:它把深度学习与物理中的重整化群联系起来,可以作为这篇信息瓶颈分析之前的背景阅读。
- 1
同样从理论角度解释深度网络为何能逐层提取相关特征,可作为信息瓶颈视角的对照背景。
读变分重整化群与 RBM 堆叠之间映射的核心想法:逐层粗粒化保留相关信息。
- 2Deep Learning and the Information Bottleneck Principle
目标论文:用信息瓶颈原理量化各层与输入、输出的互信息,并推导泛化界。
读把 DNN 表示为层与输入输出之间的互信息,信息瓶颈的最优极限以及有限样本泛化界。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“For an information theoretic perspective of this relation between deep learning and RG, refer to this fascinating paper by Naftali Tishby and Noga Zaslavsky, that brings together ideas from different fields: Deep Learning and the Information Bottleneck Principle”
“but some of his key claims are probably contradicted by empirical results.”
“In particular, I've spent the last several months in trying to understand the following two papers in increasing levels of depth”
“It's likely going to be a Tishby-like (the classic paper from 2015”