按年被提到
20232023 · 7 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2310.02989 · stat.ML

xVal: A Continuous Numerical Tokenization for Scientific Language Models

xVal:面向科学语言模型的连续数字分词
Siavash Golkar, Mariel Pettee, Michael Eickenberg, Alberto Bietti, Miles Cranmer, Geraud Krawezik, Francois Lanusse, Michael McCabe, Ruben Ohana, Liam Parker, Bruno Régaldo-Saint Blancard, Tiberiu Tesileanu 等 14 人 · 2023

用连续数值嵌入编码数字,改善语言模型处理科学数据

开始阅读 ↗PDFalphaXiv 讨论
被提及 · 5 人首次 2023-10-18 · 最近 2024-10-23
5不同的人
5活跃月份
9讨论串
25被引(OpenAlex)

LLM 对数字采用离散编码,难以处理数值密集的科学数据。xVal 提出把数字连续地编码进语言模型,使其对科学应用有更合适的归纳偏置。作者在多个文本形式的科学数据集上从头训练经过改动的语言模型,发现 xVal 在分布外泛化和计算效率等指标上总体优于其他常见的数字分词方式。

HN 为什么提它

HN 评论者在谈“LLM 算术的最大障碍是分词”时反复引用它,说明去掉分词带来的限制后,数字处理能力会好得多。

方法难度 3 / 5约 35 分钟经得起时间
读之前最好知道
1语言模型的分词(tokenization)
2Transformer 与词嵌入
3科学数据的数值表示

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
引用lawlessone · 2023-10

“The paper is here xVal”

贴出 xVal 论文原文,用连续数字嵌入服务科学语言模型
Single Digit tokenization improves LLM math abilities by up to 70x原评论 ↗
引用og_kalu · 2023-11

“It's better (GPT-4) than you could ever manage without an external tool. and that's after being severely hampered by tokenization.”

说明 GPT-4 即使被分词严重拖累,算术仍比人不借工具强,xVal 用连续编码数字
Telling GPT-4 you're scared or under pressure improves performance原评论 ↗
引用og_kalu · 2023-11

“In fact, despite being severely hobbled by Tokenization, it can perform arithmetic better than you could without a dedicated tool or pad.”

拿它说明分词严重拖累算术,但模型仍比人不借工具算得好
Transformers struggle with generalizing tasks beyond pre-training data原评论 ↗
引用og_kalu · 2023-11

“The biggest problem with LLM arithmetic is tokenization. xVal”

引用它说明分词是 LLM 算术的最大障碍
Transformers struggle with generalizing tasks beyond pre-training data原评论 ↗
引用og_kalu · 2023-11

“Unsurprisingly, removing that handicap resulted in much better arithmetic abilities. xVal”

引用它说明去掉分词这个障碍后,算术能力明显提升
Sutskever: OpenAI board doing its mission to build AGI that benefits all原评论 ↗
引用kelseyfrog · 2023-11

“Generally true before xVal[1], but not exactly true after.”

指出此前的说法大体成立,但有了 xVal 之后就不尽然
Sam Altman's ouster was precipitated by letter to board about AI breakthrough原评论 ↗
看全部 10 条 →
引用famouswaffles · 2024-03

“xVal: A Continuous Number Encoding for Large Language Models - xVal”

贴出 xVal 论文,作为逐位分词之外的连续数字编码方案
Integer Tokenization Is Insane (2023)原评论 ↗
引用thethirdone · 2024-03

“I have seen this paper before, but hadn't payed attention to the p10 vs p100 analysis.”

说之前见过这篇论文,却没留意其中 p10 与 p100 的对比分析
Integer Tokenization Is Insane (2023)原评论 ↗
引用famouswaffles · 2024-09

“fundamental inability ? No.”

贴出 xVal 连续数字分词的论文,来反驳 LLM 算术存在根本缺陷的说法
AGI is far from inevitable原评论 ↗
引用kelseyfrog · 2024-10

“Papers like xVal[1], point toward specialization strategies in tokenization.”

拿 xVal 作为分词专门化策略的例子
Probably pay attention to tokenizers原评论 ↗
被提交为 story · 1 次