“Link to previous paper: https://arxiv.org/abs/2308.14711 An attempt at a summary: They use a sigmoid function to make differentiable "soft" branches, and stack them to construct a binary tree, with the goal of only taking one branch at inference time (but training the whole tree) leading to log(W) instead of W inference cost. They gradually harden the b…”
按年被提到
20232023 · 3 次2026
Fast Feedforward Networks
Peter Belcak, Roger Wattenhofer · 2023
被提及 · 3 人首次 2023-09-20 · 最近 2023-11-22
3不同的人
2活跃月份
3讨论串
10被引(OpenAlex)
arXiv 摘要(候选层不写导读)
We break the linear link between the layer size and its inference cost by introducing the fast feedforward (FFF) architecture, a log-time alternative to feedforward networks. We demonstrate that FFFs are up to 220x faster than feedforward networks, up to 6x faster than mixture-of-experts networks, and exhibit better training properties than mixtures of experts thanks to noiseless conditional execution. Pushing FFFs to the limit, we show that they can use as little as 1% of layer neurons for inference in vision transformers while preserving 94.2% of predictive performance.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及评论WithinReason · 2023-11
被提交为 story · 2 次
- Fast Feedforward Networks ↗jasondavies · 2023-09-20 · 1 分
- Fast Feedforward Networks, 220x faster than feedforward networks ↗reqo · 2023-09-21 · 2 分