推荐airgapstopgap · 2023-06
“Interesting release from Apple but there's been a similar (and I think more promising architecturally) paper from Meta lately:”
推荐 Meta 的这篇论文,认为架构上比苹果的发布更有前景
多尺度解码器架构,让无分词的字节级模型处理百万字节序列
MEGABYTE 把序列切成 patch,patch 内用局部子模型、patch 之间用全局模型,实现次二次方的自注意力、同等算力下更大的前馈层,以及更好的解码并行度。实验显示字节级模型在长上下文语言建模上可与子词模型竞争,在 ImageNet 密度估计上达到当时最优,并能直接对原始音频文件建模,说明无分词的自回归序列建模在规模上是可行的。
HN 上的人在讨论“能否去掉分词器”时,把它当作最有前景的无分词(tokenization-free)LLM 架构来推荐,并用它与后续的字节级工作对比。
“Interesting release from Apple but there's been a similar (and I think more promising architecturally) paper from Meta lately:”
“The most effective architecture for LLMs without tokenization is Meta's MEGABYTE architecture: MEGABYTE.”
“Paper: MEGABYTE”
“That's been on the table e.g. [0][1].”
“For text, there are a few papers that train the tokenization and language model end-to-end, see: MEGABYTE”