按年被提到
20232023 · 1 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2304.08485 · cs.CV

Visual Instruction Tuning

Haotian Liu, Chunyuan Li, Qingyang Wu, Yong Jae Lee · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-12-07 · 最近 2024-05-15
2不同的人
2活跃月份
2讨论串
1,261被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Instruction tuning large language models (LLMs) using machine-generated instruction-following data has improved zero-shot capabilities on new tasks, but the idea is less explored in the multimodal field. In this paper, we present the first attempt to use language-only GPT-4 to generate multimodal language-image instruction-following data. By instruction tuning on such generated data, we introduce LLaVA: Large Language and Vision Assistant, an end-to-end trained large multimodal model that connects a vision encoder and LLM for general-purpose visual and language understanding.Our early experiments show that LLaVA demonstrates impressive multimodel chat abilities, sometimes exhibiting the behaviors of multimodal GPT-4 on unseen images/instructions, and yields a 85.1% relative score compared with GPT-4 on a synthetic multimodal instruction-following dataset. When fine-tuned on Science QA, the synergy of LLaVA and GPT-4 achieves a new state-of-the-art accuracy of 92.53%. We make GPT-4 generated visual instruction tuning data, our model and code base publicly available.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论og_kalu · 2023-12

“>the sounds for man, and the word man can be mapped. The concept of man can be mapped from English to Chinese. Given enough of these clues, and a large enough data set it's not shocking that models get decent at inference between languages. 1. Many such concepts cannot be directly mapped between different languages especially with distant language pairs.…”

Vowels and Diphthongs in Sperm Whales原评论 ↗
评论llama_person · 2024-05

“Here's a tutorial https://wandb.ai/byyoung3/ml-news/reports/How-to-Fine-Tune-LLaVA-on-a-Custom-Dataset--Vmlldzo2NjUwNTc1 There's not really a super easy to use software solution yet, but a few different ones have cropped up. Right now you'll have to read papers to get the training recipes. - https://github.com/haotian-liu/LLaVA/blob/main/scripts/finetun…”

PaliGemma: Open-Source Multimodal Model by Google原评论 ↗