按年被提到
20242024 · 3 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2405.00732 · cs.CL

LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, A Technical Report

Justin Zhao, Timothy Wang, Wael Abid, Geoffrey Angus, Arnav Garg, Jeffery Kinnison, Alex Sherstinsky, Piero Molino, Travis Addair, Devvret Rishi · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2024-05-03 · 最近 2024-07-01
3不同的人
2活跃月份
3讨论串
27被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Low Rank Adaptation (LoRA) has emerged as one of the most widely adopted methods for Parameter Efficient Fine-Tuning (PEFT) of Large Language Models (LLMs). LoRA reduces the number of trainable parameters and memory usage while achieving comparable performance to full fine-tuning. We aim to assess the viability of training and serving LLMs fine-tuned with LoRA in real-world applications. First, we measure the quality of LLMs fine-tuned with quantized low rank adapters across 10 base models and 31 tasks for a total of 310 models. We find that 4-bit LoRA fine-tuned models outperform base models by 34 points and GPT-4 by 10 points on average. Second, we investigate the most effective base models for fine-tuning and assess the correlative and predictive capacities of task complexity heuristics in forecasting the outcomes of fine-tuning. Finally, we evaluate the latency and concurrency capabilities of LoRAX, an open-source Multi-LoRA inference server that facilitates the deployment of multiple LoRA fine-tuned models on a single GPU using shared base model weights and dynamic adapter loading. LoRAX powers LoRA Land, a web application that hosts 25 LoRA fine-tuned Mistral-7B LLMs on a single NVIDIA A100 GPU with 80GB memory. LoRA Land highlights the quality and cost-effectiveness of employing multiple specialized LLMs over a single, general-purpose LLM.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论w4nderlust · 2024-07

“We got very similar findings: we published a paper that show that smaller LLMs (3-7b) when finetuned with LoRA can match or outperform GPT-4 on a variety of tasks (29 out of 31) including classification, summarization, info extraction, "reasoning". https://arxiv.org/abs/2405.00732 (Predibase cofounder and coauthor of the paper)”

My finetuned models beat OpenAI's GPT-4原评论 ↗
被提交为 story · 2 次