“From a practical standpoint, the main citation's is pretty informative:”
What's Hidden in a Randomly Weighted Neural Network?
随机权重网络里藏着无需训练权重就表现良好的子网络
训练神经网络通常等同于学习权重值,作者则证明随机权重的网络中存在无需训练权重就有出色表现的子网络:在随机权重的 Wide ResNet-50 中,存在一个比 ResNet-34 更小、在 ImageNet 上与已训练 ResNet-34 表现相当的(随机权重)子网络。作者还给出了有效找到这些「未训练子网络」的算法,并通过实验表明随着固定权重的随机网络变宽变深,其未训练子网络的精度逐渐接近权重已学习的网络。
HN 上的人引用它来说明在随机网络中通过剪枝找出好用子网络的思路(权重不经训练),也有人认为这只说明参数空间极其庞大。
学习路径
读这篇之前,按顺序读书架上的 4 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出先熟悉残差网络与初始化,再看彩票假设(稀疏子网络的存在),最后读目标论文如何在随机权重网络里直接找出可用子网络。
- 1
- 2
- 3
目标论文的随机权重需要合适的初始化方案,He 初始化是其基础。
读初始化的方差设置。
- 4
彩票假设说明稠密网络中存在稀疏可训练子网络,目标论文把它推进到无需训练权重的子网络。
读剪枝并回到初始权重重新训练的流程与发现。
- 5What's Hidden in a Randomly Weighted Neural Network?
目标论文:随机权重网络里存在不需训练权重就表现很好的子网络,并给出寻找它们的算法。
读edge-popup 算法如何给边打分并选取;实验中网络宽度与子网络性能的关系。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Seems relevant to what you're working on.”
“The fact that a dense neural network with 20M parameters performs equally well as a model with 20M random values and 20M _bits_ worth of parameters means nothing more than that the parameter space is ridiculously large.”
“As shown in [1] that can lead to randomly initialized networks working well, too,”
“What they prove in this paper is the stronger version”
“As for the random network, I was referring to What's Hidden in a Randomly Weighted Neural Network? and I have seen similar effects in my own work”
看全部 7 条 →
“They were able to find a subnetwork in a randomly weighted network with equivalent performance to a trained network of a much smaller size.”
- What's Hidden in a Randomly Weighted Neural Network? ↗hardmaru · 2019-12-03 · 4 分
- What's Hidden in a Randomly Weighted Neural Network? ↗asdfadf123123 · 2019-12-04 · 3 分
- What's Hidden in a Randomly Weighted Neural Network? ↗dstein64 · 2020-04-24 · 2 分