引用CGamesPlay · 2024-05
“This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains.”
指出这是 FNet 论文,被数百篇引用,部分涉及效率提升
用傅里叶变换替代自注意力,达到 BERT 92–97% 精度且训练更快。
论文表明可以用简单的线性变换“混合”输入 token 来替换 Transformer 编码器中的自注意力子层。用不带参数的标准傅里叶变换替换后,在 GLUE 上能达到 BERT 对应模型 92–97% 的精度,在 GPU 上训练快 80%、TPU 上快 70%(512 长度)。在 Long Range Arena 上,FNet 精度与最准的高效 Transformer 相当,且在各序列长度下更快,内存占用也小。
评论者把它当作“用 FFT 替代自注意力、以少许精度换取更低算力和能耗”的例子,在讨论高效注意力替代方案时引用。
“This is the paper, which appears to be cited in hundreds of others, some of which appear to be about efficiency gains.”
“"Fnet: Mixing tokens with fourier transforms" (2021) FNet”
“Google introduced this idea in 2022 with "FNet: Mixing Tokens with Fourier Transforms" [0].”
“1. Take FNet.”
“self-attention can be replaced with FFT for a loss in accuracy and a reduction in kWh”