按年被提到
20232023 · 3 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2312.08361 · cs.LG

Distributed Inference and Fine-tuning of Large Language Models Over The Internet

Alexander Borzunov, Max Ryabinin, Artem Chumachenko, Dmitry Baranchuk, Tim Dettmers, Younes Belkada, Pavel Samygin, Colin Raffel · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 4 人首次 2023-12-14 · 最近 2024-01-02
4不同的人
2活跃月份
4讨论串
36被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuning of LLMs, comparing local and distributed strategies. We observe that a large enough model (50B+) can run efficiently even on geodistributed devices in a consumer-grade network. This could allow running LLM efficiently by pooling together idle compute resources of multiple research groups and volunteers. We address two open problems: (1) how to perform inference and fine-tuning reliably if any device can disconnect abruptly and (2) how to partition LLMs between devices with uneven hardware, joining and leaving at will. In order to do that, we develop special fault-tolerant inference algorithms and load-balancing protocols that automatically assign devices to maximize the total system throughput. We showcase these algorithms in Petals - a decentralized system that runs Llama 2 (70B) and BLOOM (176B) over the Internet up to 10x faster than offloading for interactive generation. We evaluate the performance of our system in simulated conditions and a real-world setup spanning two continents.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

没有评论直接链到它;下面是它被提交的讨论串里回复最多的评论(不计入统计)
讨论串评论jmrjmr · 2023-12

“Abstract Large language models (LLMs) are useful in many NLP tasks and become more capable with size, with the best open-source models having over 50 billion parameters. However, using these 50B+ models requires high-end hardware, making them inaccessible to most researchers. In this work, we investigate methods for cost-efficient inference and fine-tuni…”

Distributed Inference and Fine-Tuning of Large Language Models over the Internet原评论 ↗
讨论串评论knightoffaith · 2024-01

“Great that folks are starting to think about this, I think it's important to keep the power of AI decentralized. Thought it seems like for this to be practical, they need to flesh out the Security section in the appendix to deal with bad actors. I like the probabilistic approach in lieu of full byzantine fault tolerance. Would be interesting to see what k…”

Distributed Inference and Fine-Tuning of Large Language Models over the Internet原评论 ↗
讨论串评论bugglebeetle · 2024-01

“A distributed network of GPUs for something other than funny-money sudoku puzzle-solving is a nice development.”

Distributed Inference and Fine-Tuning of Large Language Models over the Internet原评论 ↗
被提交为 story · 4 次