推荐naasking · 2025-06
“Large Concept Models is promising for this reason:”
认为大概念模型有前景:句子级概念表示可统一不同模态
在 SONAR 句子嵌入空间里自回归预测下一个句子的模型
论文尝试让模型在显式的高层语义表示“概念”上运作,而不是逐 token 处理;作为可行性验证,假定一个概念对应一个句子,并使用支持 200 种语言的文本与语音 SONAR 句子嵌入空间。大概念模型在该空间里做自回归的句子预测,探索了 MSE 回归、多种扩散式生成和量化 SONAR 空间三类做法,先用 1.6B 参数模型试验,再把一种架构扩展到 7B 参数。在摘要和新任务“摘要扩写”上评测,并展示了对多种语言的零样本泛化。
评论者把它当作句子级隐空间自回归、语言与模态无关表示这类新架构的例子来引用。
“Large Concept Models is promising for this reason:”
“I think probably an example of the kind of new architectures he supports is FB's Large Concept Models [1].”
“this is related to the large concept model: Large Concept Models, where the latent space is SONAR, which is very much designed for this purpose.”