引用woodson · 2025-05
“How does this differ from SuperBPE, which seems to pursue a similar goal? SuperBPE”
问这个工作与目标相似的 SuperBPE 有何不同
让 BPE 先学子词再学跨空格的超词,token 更少、效果更好
论文质疑几乎所有语言模型分词方案默认 token 为子词(不跨词边界)是否限制了模型,因为空格并非可靠的语义边界。他们提出 SuperBPE 分词器,在 BPE 中加入简单的预分词课程:先学子词,再学跨越空格的超词。词表固定为 200k 时,SuperBPE 编码同一文本平均最多少 33% 的 token;固定模型规模、词表和训练算力、只改词表学习算法来预训练 8B 模型,在 30 个下游任务上平均比 BPE 基线绝对提高 4.0%(MMLU 提高 8.2%),推理算力减少 27%。
评论者把它当作目标相近的并行工作来对比,并指出数字分组等效果可以只靠修改预分词正则实现,不必显式加逗号。
“How does this differ from SuperBPE, which seems to pursue a similar goal? SuperBPE”
“Both SuperBPE and [OTHER_PAPER] (co-author) both independently noted that you can do this with just by modifying the pre-tokenization regex, without having to explicitly add commas.”