推荐og_kalu · 2023-04
“There's something a bit more mindblowing than that.”
推荐它,说比这更令人震惊:语言与视觉模型表示相似,线性映射就够
单个线性映射就能把视觉特征接入冻结的语言模型
论文检验一个较强的假设:冻结的纯文本语言模型与纯视觉模型学到的概念表示足够相似,只需一个线性映射就能连接。做法是只训练一层线性投影,把视觉模型的图像表示作为连续提示输入冻结的语言模型,在图像描述和视觉问答上与同时微调图像编码器和文本解码器的模型(如 MAGMA)相比有竞争力。对 BEIT、NF-ResNet、CLIP 三种编码器的比较表明,它们传递视觉属性信息的效果相当,但带语言监督的编码器更能凸显类别信息,因此在基准上明显更好。
评论者用它说明语言模型与视觉模型的表征出人意料地相似,只需一层线性投影就能把二者连起来。
“There's something a bit more mindblowing than that.”
“But there are interesting results from just a few days ago saying that text by itself is also enough:”
“Just found this paper that demonstrated this a few months ago”
“I've been reading some stuff[1][2][3] about smaller models, and my impression has been that the latent space is shockingly linear for those ones.”
“You can connect linguistic and visual models with a single linear projection layer. That's it.”