“If you're interested in the generalization ability of neural networks I can recommend the following paper:”
Intriguing properties of neural networks
发现不可察觉的微小扰动即可让神经网络误分类
论文报告了深度神经网络的两个反直觉性质。其一,按多种单元分析方法,高层的单个单元与单元的随机线性组合并无区别,说明语义信息存在于整个空间而非单个单元中;其二,网络学到的输入输出映射在相当程度上是不连续的:对图像施加通过最大化预测误差找到的、人眼不可察觉的扰动,就能让网络误分类,而且这类扰动并非学习的随机产物,同一个扰动还能让在不同数据子集上训练的另一个网络对同一输入误分类。
HN 上的人把它当作对抗样本的原始论文,常拿它说明深度网络脆弱、扰动可跨网络迁移,并在讨论自动驾驶等场景的鲁棒性时引用。
学习路径
读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮助理解单元分析的背景:无监督学习出的高层特征、卷积网络的可视化与 dropout,再读神经网络的两个反直觉性质。
- 1
论文的单元分析沿用了寻找使单个神经元激活最大的输入这类做法。
读抓如何通过最大激活输入分析单个神经元。
- 2
提供卷积网络内部单元可视化的另一种方法,与论文的单元分析相关。
读抓反卷积式可视化,理解单个单元对应的图像模式。
- 3
dropout 是论文研究的网络训练方法之一,需要知道其作用。
读抓随机丢弃单元的做法;实验部分略读。
- 4Intriguing properties of neural networks
发现不可察觉的微小扰动即可让神经网络误分类,且单元与单元的随机线性组合没有区别。
读抓第一个性质(空间而非单元携带信息)和第二个性质(对抗样本)的实验。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Naturally the paper is where the good stuff is.”
“Don't forget "Intriguing properties of neural networks", otherwise known as "Does Deep Learning have deep flaws?".”
“start here: Intriguing properties of neural networks”
“a recent paper [1] shows that if we feed to an object classifying NN an image with crafted but unnoticeable perturbations the predictions change completely.”
“This paper shows that one can craft noise which will fool a neural network. But it isn't noise that is found in real life.”
看全部 20 条 →
“Witness, for example, all the recent "we can fool deep learning image recognition systems" papers.”
“To be clear I meant this paper as the original paper for adversarial images.”
“Even if there is no overfitting, deep neural networks seem to learn a surprisingly discontinuous function.”
“has examples where you get radically different classifications that I don't think would occur naturally or survive a blur with some random element”
“AI/ML algos cannot magically guarantee safety, for example, deep nets are known to be very un-robust: Intriguing properties of neural networks.”
“For example, self-driving cars based on machine learning may be susceptible to adversarial attacks that cause the car to behave unpredictably [1,2].”
“From one of the canonical papers on adversarial images:”
“See Intriguing properties of neural networks and [OTHER_PAPER]”
“We can cause the network to misclassify an image by applying a certain imperceptible perturbation, which is found by maximizing the network's prediction error.”
“Adversarial examples have been shown to, for instance, be transferable across different networks with different hyperparameters (e.g., number of layers) trained on disjoint subsets of a training set [0, section 4.2].”
“Szegedy et al [9] has the following abstract:”
“is a major research topic at the moment.”
“This bus is an ostrich:”
“adversarial examples [2],”