“Good overview. At the other extreme, some recent works [1,2] show why it’s sometime better to scale down instead of up, especially for some humanlike capabilities like generalization: [1] https://direct.mit.edu/tacl/article/doi/10.1162/tacl_a_00489/112499/Minimum-Description-Length-Recurrent-Neural [2] https://arxiv.org/abs/1906.04358”
Weight Agnostic Neural Networks
Not all neural network architectures are created equal, some perform much better than others for certain tasks. But how important are the weight parameters of a neural network compared to its architecture? In this work, we question to what extent neural network architectures alone, without learning any weight parameters, can encode solutions for a given task. We propose a search method for neural network architectures that can already perform a task without any explicit weight training. To evaluate these networks, we populate the connections with a single shared weight parameter sampled from a uniform random distribution, and measure the expected performance. We demonstrate that our method can find minimal neural network architectures that can perform several reinforcement learning tasks without weight training. On a supervised learning domain, we find network architectures that achieve much higher than chance accuracy on MNIST using random weights. Interactive version of this paper at https://weightagnostic.github.io/
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“There is a great paper, Weight Agnostic Neural Networks [0], that explores this topic. They experiment with using a single shared weight for a network while using an evolutionary algorithm to find architectures that are themselves biased towards being effective on specific problems. The upshot is that once you've found an architecture that is already bia…”
“> While playing around with this setup, I tried re-training the network with the activation function for the first layer replaced with sin(x) and it ends up working pretty much the same way. There is some evidence that the activation functions and weights can be arbitrarily selected assuming you have a way to evolve the topology of the network. https://…”