按年被提到
20242024 · 1 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2410.05864 · cs.CL

From Tokens to Words: On the Inner Lexicon of LLMs

Guy Kaplan, Matanel Oren, Yuval Reif, Roy Schwartz · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2024-10-23 · 最近 2025-02-25
2不同的人
2活跃月份
2讨论串
13被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Natural language is composed of words, but modern large language models (LLMs) process sub-words as input. A natural question raised by this discrepancy is whether LLMs encode words internally, and if so how. We present evidence that LLMs engage in an intrinsic detokenization process, where sub-word sequences are combined into coherent whole-word representations at their last token. Our experiments show that this process primarily takes place within the early and middle layers of the model. We further demonstrate its robustness to arbitrary splits (e.g., "cats" to "ca" and "ts"), typos, and importantly-to out-of-vocabulary words: when feeding the last token internal representations of such words to the model as input, it can "understand" them as the complete word despite never seeing such representations as input during training. Our findings suggest that LLMs maintain a latent vocabulary beyond the tokenizer's scope. These insights provide a practical, finetuning-free application for expanding the vocabulary of pre-trained models. By enabling the addition of new vocabulary words, we reduce input length and inference iterations, which reduces both space and model latency, with little to no loss in model accuracy.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论Lerc · 2025-02

“So if they can come up with their own internal representations is that sufficient? https://arxiv.org/abs/2410.05864 Or are you wanting them to use keys that are not on their keyboard?”

Google Co-Scientist AI fed previous paper with the answer in it原评论 ↗
被提交为 story · 1 次