按年被提到
20242024 · 3 次2026
大模型与语言模型›大模型推理与思维链·arXiv:2402.14903 · cs.CL

Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs

Aaditya K. Singh, DJ Strouse · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2024-03-02 · 最近 2025-06-24
5不同的人
2活跃月份
3讨论串
47被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Tokenization, the division of input text into input tokens, is an often overlooked aspect of the large language model (LLM) pipeline and could be the source of useful or harmful inductive biases. Historically, LLMs have relied on byte pair encoding, without care to specific input domains. With the increased use of LLMs for reasoning, various number-specific tokenization schemes have been adopted, with popular models like LLaMa and PaLM opting for single-digit tokenization while GPT-3.5 and GPT-4 have separate tokens for each 1-, 2-, and 3-digit numbers. In this work, we study the effect this choice has on numerical reasoning through the use of arithmetic tasks. We consider left-to-right and right-to-left tokenization for GPT-3.5 and -4, finding that right-to-left tokenization (enforced by comma separating numbers at inference time) leads to largely improved performance. Furthermore, we find that model errors when using standard left-to-right tokenization follow stereotyped error patterns, suggesting that model computations are systematic rather than approximate. We show that the model is able to convert between tokenizations easily, thus allowing chain-of-thought-inspired approaches to recover performance on left-to-right tokenized inputs. We also find the gap between tokenization directions decreases when models are scaled, possibly indicating that larger models are better able to override this tokenization-dependent inductive bias. In summary, our work performs the first study of how number tokenization choices lead to differences in model performance on arithmetic tasks, accompanied by a thorough analysis of error patterns. We hope this work inspires practitioners to more carefully ablate number tokenization-related choices when working towards general models of numerical reasoning.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论famouswaffles · 2024-03

“>It has seemed to me that the GPT would be considerably better at numbers if it just considered each digit as a token. Has anyone actually done an experiment to test this? Yeah Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 xVal: A Continuous Number Encoding for Large Language Models - …”

Integer Tokenization Is Insane (2023)原评论 ↗
评论thethirdone · 2024-03

“> Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs - https://arxiv.org/abs/2402.14903 Very interesting paper. It does make sense to me the R2L chunking would be better than L2R chunking. It doesn't actually study single digit tokenization. I am mostly interested in a direct comparison between an LLM wide tokenization vs sin…”

Integer Tokenization Is Insane (2023)原评论 ↗
评论cschmidt · 2025-06

“This paper has a good solution: https://arxiv.org/abs/2402.14903 You right to left tokenize in groups of 3, so 1234567 becomes 1 234 567 rather than the default 123 456 7. And if you ensure all 1-3 digits groups are in the vocab, it does much better. Both https://arxiv.org/abs/2503.13423 and https://arxiv.org/abs/2504.00178 (co-author) both independent…”

The bitter lesson is coming for tokenization原评论 ↗
评论ijk · 2025-06

“The math tokenization research is probably closest. GPT-2 tokenization was a demonstratable problem: https://www.beren.io/2023-02-04-Integer-tokenization-is-insane/ (Prior HN discussion: https://news.ycombinator.com/item?id=39728870 ) More recent research: https://huggingface.co/spaces/huggingface/number-tokenization-blog Tokenization counts: the impa…”

The bitter lesson is coming for tokenization原评论 ↗
被提交为 story · 1 次