引用f38zf5vdt · 2023-03
“The multimodal models for text-vision were all established using a scaffold of architectures that unified text-token and vision-token similarity e.g. BLIP2.”
拿 BLIP2 为例,说明多模态模型都建立在统一文本与视觉 token 相似度的架构上
用轻量 Q-Former 连接冻结的图像编码器与冻结的大语言模型
论文提出 BLIP-2,一种通用且高效的视觉语言预训练策略:复用现成的冻结图像编码器和冻结大语言模型,通过轻量的 Querying Transformer(Q-Former)弥合模态差距,并分两阶段预训练。BLIP-2 在多项视觉语言任务上达到最优表现,而可训练参数大幅减少,例如零样本 VQAv2 上比 Flamingo80B 高 8.7%,可训练参数少 54 倍。模型还展示了按自然语言指令进行零样本图像到文本生成的能力。
评论者把它当作多模态模型架构的代表,并指出 MiniGPT-4 直接复用了整个 BLIP-2 编码器。
“The multimodal models for text-vision were all established using a scaffold of architectures that unified text-token and vision-token similarity e.g. BLIP2.”
“If you look at their code it's literally using the entire BLIP-2 encoder (Salesforce code).”
“Given the exposure hn has had to blip-2 it might be good to provide more context.”