“The paper is here xVal”
xVal: A Continuous Numerical Tokenization for Scientific Language Models
用连续数值嵌入编码数字,改善语言模型处理科学数据
LLM 对数字采用离散编码,难以处理数值密集的科学数据。xVal 提出把数字连续地编码进语言模型,使其对科学应用有更合适的归纳偏置。作者在多个文本形式的科学数据集上从头训练经过改动的语言模型,发现 xVal 在分布外泛化和计算效率等指标上总体优于其他常见的数字分词方式。
HN 评论者在谈“LLM 算术的最大障碍是分词”时反复引用它,说明去掉分词带来的限制后,数字处理能力会好得多。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“It's better (GPT-4) than you could ever manage without an external tool. and that's after being severely hampered by tokenization.”
“In fact, despite being severely hobbled by Tokenization, it can perform arithmetic better than you could without a dedicated tool or pad.”
“The biggest problem with LLM arithmetic is tokenization. xVal”
“Unsurprisingly, removing that handicap resulted in much better arithmetic abilities. xVal”
“Generally true before xVal[1], but not exactly true after.”
看全部 10 条 →
“xVal: A Continuous Number Encoding for Large Language Models - xVal”
“I have seen this paper before, but hadn't payed attention to the p10 vs p100 analysis.”
“fundamental inability ? No.”
“Papers like xVal[1], point toward specialization strategies in tokenization.”
- Single Digit Tokenization improves LLM Math abilities by up to 70x ↗og_kalu · 2023-10-18 · 1 分