推荐marmaduke · 2024-01
“The rwkv4 paper is quite detailed and has examples of prompt and responses on the last few pages”
推荐 RWKV4 论文:内容相当详细,最后几页还有提示和回答的例子
RWKV:训练像 Transformer 并行、推理像 RNN 恒定开销的架构
提出 RWKV(Receptance Weighted Key Value)架构,用线性注意力机制,使模型既可写成 Transformer 形式并行训练,又可写成 RNN 形式,推理时计算和内存开销保持恒定。作者把模型扩大到 140 亿参数,称是迄今训练过的最大稠密 RNN,性能与同规模 Transformer 相当。
HN 上的人把它当作 RNN 类架构可能是长上下文下一步的代表例子,也引用它自己的局限性一节来讨论长上下文回忆能力。
“The rwkv4 paper is quite detailed and has examples of prompt and responses on the last few pages”
“From my understanding some form of RNNs are likely to be the next step for longer context. See RWKV as an example of a decent RNN RWKV”
“Here is a relevant tidbit from the RWKV paper "Limitations" section:”
“* RWKV: [OTHER_PAPER] / [OTHER_PAPER] RWKV”