“Abstract Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuni…”
Distributed Inference and Fine-tuning of Large Language Models Over The Internet
Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals - a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)“Great that folks are starting to think about this, I think it's important to keep the power of AI decentralized. Thought it seems like for this to be practical, they need to flesh out the Security section in the appendix to deal with bad actors. I like the probabilistic approach in lieu of full byzantine fault tolerance. Would be interesting to see what k…”
“A distributed network of GPUs for something other than funny-money sudoku puzzle-solving is a nice development.”
- Distributed Inference and Fine-Tuning of Large Language Models over the Internet ↗jmrjmr · 2023-12-14 · 1 分
- Distributed Inference and Fine-Tuning of Large Language Models over the Internet ↗Brajeshwar · 2023-12-15 · 1 分
- Distributed Inference and Fine-Tuning of Large Language Models over the Internet ↗milliondreams · 2023-12-17 · 2 分
- Distributed Inference and Fine-Tuning of Large Language Models over the Internet ↗cyanf · 2024-01-02 · 102 分