“Check out the visual IQ test results in the paper.”
Language Is Not All You Need: Aligning Perception with Language Models
微软 Kosmos-1:能看图、能上下文学习的多模态大语言模型
论文提出 Kosmos-1,一个在网页规模的图文交错语料、图文对和纯文本上从零训练的多模态大语言模型(MLLM),无需梯度更新即可在零样本、少样本和多模态思维链提示下完成任务。它在语言理解与生成、OCR-free 的文档图像理解、图像描述、视觉问答和带文字说明的图像识别上都有不错表现,并表明语言与多模态之间能够互相迁移知识。论文还提出了一个 Raven IQ 测试数据集,用来考察模型的非语言推理能力。
GPT-4 发布前后,HN 上的人拿它当例子,说明图像可以编码成向量再映射到 token 空间、与文本一起输入语言模型。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Check the kosmos-1 paper from Microsoft that appeared a few days before GPT4 was released: Language Is Not All You Need”
“Edit: Ah, looks like this is the link to the paper: Language Is Not All You Need”
“(see eg PIXEL [OTHER_PAPER] or Kosmos Language Is Not All You Need or [OTHER_PAPER] [OTHER_PAPER] [OTHER_PAPER] )”
“"Language Is Not All You Need: Aligning Perception with Language Models"”
“combine a similar encoding as that of Kosmos-1[1]”
看全部 8 条 →
“They probably use something similar to Kosmos-1: Encode images as vectors with something like CLIP, then map them to the token space”
“can do pretty amazing things, and (iirc) with better days efficiency than a text only model.”
- LLMs that can see and hear ↗craftsquick · 2023-02-28 · 2 分