“Figure 4 is a really good example of why byte-level models are wasting computation.”
Byte Latent Transformer: Patches Scale Better Than Tokens
按下一字节熵动态分补丁的字节级大模型,无需分词器
Byte Latent Transformer(BLT)是一种字节级 LLM 架构,把字节按下一字节的熵动态切成不定长补丁作为计算单元,在数据可预测时用长补丁,在复杂处分配更多算力。论文做了首个 FLOP 受控的字节级模型扩展研究,规模达 8B 参数、4T 训练字节,称其首次在大规模下追平基于分词的模型,同时提升推理效率和鲁棒性。在固定推理成本下,同时增大补丁和模型尺寸可获得比分词模型更好的扩展性。
HN 上的人在讨论分词缺陷(如拼写、字符级任务)时引用它,把它当作无需 BPE 分词的代表方案,也有人批评它重新引入缓存和 n-gram。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Have you seen the Byte-latent Transformer paper?”
“In the original Byte Latent Transformer paper they reintroduce ugly caching and n-grams which I'm looking to eliminate.”
“was good but didnt turn up Byte Latent Transformer <- which is more recent, more people might want”
“Interesting recent research from Meta going in what I would call the opposite direction from an interpretability standpoint: Byte Latent Transformer”
“Also, BLT[2] from Meta and ByteFormer[3] from Apple.”
看全部 9 条 →
“See the Byte Latent Transformer - Byte Latent Transformer - where a similar scale byte latent model trained on the same dataset >>> a vanilla transformer on word and character manipulation tasks.”
“It seems that it can be fixed by simply doing away with Byte Pair Encoding tokenization.”
“Byte Latent Transformers (BLT): Byte Latent Transformer”
- Byte latent transformer: Patches scale better than tokens (2024) ↗dlojudice · 2025-05-12 · 107 分