按年被提到
20242024 · 2 次2026
大模型与语言模型›模型表达力与计算理论·arXiv:2410.23168 · cs.LG

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters

Haiyang Wang, Yue Fan, Muhammad Ferjad Naeem, Yongqin Xian, Jan Eric Lenssen, Liwei Wang, Federico Tombari, Bernt Schiele · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-10-31 · 最近 2024-11-01
2不同的人
2活跃月份
2讨论串
5被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Transformers have become the predominant architecture in foundation models due to their excellent performance across various domains. However, the substantial cost of scaling these models remains a significant concern. This problem arises primarily from their dependence on a fixed number of parameters within linear projections. When architectural modifications (e.g., channel dimensions) are introduced, the entire model typically requires retraining from scratch. As model sizes continue growing, this strategy results in increasingly high computational costs and becomes unsustainable. To overcome this problem, we introduce TokenFormer, a natively scalable architecture that leverages the attention mechanism not only for computations among input tokens but also for interactions between tokens and model parameters, thereby enhancing architectural flexibility. By treating model parameters as tokens, we replace all the linear projections in Transformers with our token-parameter attention layer, where input tokens act as queries and model parameters as keys and values. This reformulation allows for progressive and efficient scaling without necessitating retraining from scratch. Our model scales from 124M to 1.4B parameters by incrementally adding new key-value parameter pairs, achieving performance comparable to Transformers trained from scratch while greatly reducing training costs. Code and models are available at https://github.com/Haiyang-W/TokenFormer.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论cs702 · 2024-11

“The authors factorize every weight matrix with an attention mechanism: weight = attention(token_query, weight_keys, weight_values). In other words, they query weight_keys to fetch the weight_values, and mix them to compute each weight on the spot. Increasing model size becomes a matter of adding more weight_keys and weight_values, and incrementally trai…”

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters原评论 ↗
讨论串评论valine · 2024-11

“I would like to see a comparison for the inference time compute between a regular transformer and this. I’m assuming token/s is lower since you need to compute the weights of the model for each token prior to the actual attention calculations for the sequence position.”

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters原评论 ↗
讨论串评论sapphire42 · 2024-11

“As someone who has worked in this space, this paper is unfortunately total BS. Their claimed theoretical advancement is as follows. If you want to transform an input vector X to another vector Y of different dimension, "normal" people suggest to use a linear projection: create an appropriately sized matrix W and simply multiply it by your input: Given X…”

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters原评论 ↗
被提交为 story · 2 次