“I don't think this is THAT world changing. Generalist / multitask learning is already displayed in many models across orgs like Deepmind and others. This Graph NN model is performing algorithms that typically rely on graph theory. It's unsurprising that there's little friction and high success in combining embedding spaces on various graph algorithms by r…”
A Generalist Neural Algorithmic Learner
The cornerstone of neural algorithmic reasoning is the ability to solve algorithmic tasks, especially in a way that generalises out of distribution. While recent years have seen a surge in methodological improvements in this area, they mostly focused on building specialist models. Specialist models are capable of learning to neurally execute either only one algorithm or a collection of algorithms with identical control-flow backbone. Here, instead, we focus on constructing a generalist neural algorithmic learner -- a single graph neural network processor capable of learning to execute a wide range of algorithms, such as sorting, searching, dynamic programming, path-finding and geometry. We leverage the CLRS benchmark to empirically show that, much like recent successes in the domain of perception, generalist algorithmic learners can be built by "incorporating" knowledge. That is, it is possible to effectively learn algorithms in a multi-task manner, so long as we can learn to execute them well in a single-task regime. Motivated by this, we present a series of improvements to the input representation, training regime and processor architecture over CLRS, improving average single-task performance by over 20% from prior art. We then conduct a thorough ablation of multi-task learners leveraging these improvements. Our results demonstrate a generalist learner that effectively incorporates knowledge captured by specialist models.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)“This could be a piece of the AGI puzzle. Learning how to combine/integrate various kinds of ANNs trained on vastly different domains is something that will have to be done at some point. At least this seems to be a reasonable path to explore.”
“It feels like there's an interesting release from the one of the major AI labs nearly every week now. Is this a hype wave before the next AI winter, or are we accelerating towards something truly profound?”
- A Generalist Neural Algorithmic Learner ↗hardmaru · 2022-09-23 · 1 分
- A Generalist Neural Algorithmic Learner ↗tosh · 2022-12-08 · 1 分