按年被提到
20242024 · 4 次2026
强化学习与智能体›大模型智能体·arXiv:2401.02415 · cs.CL

LLaMA Pro: Progressive LLaMA with Block Expansion

Chengyue Wu, Yukang Gan, Yixiao Ge, Zeyu Lu, Jiahao Wang, Ye Feng, Ying Shan, Ping Luo · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 4 人首次 2024-01-05 · 最近 2024-11-02
4不同的人
2活跃月份
4讨论串
1被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Humans generally acquire new skills without compromising the old; however, the opposite holds for Large Language Models (LLMs), e.g., from LLaMA to CodeLLaMA. To this end, we propose a new post-pretraining method for LLMs with an expansion of Transformer blocks. We tune the expanded blocks using only new corpus, efficiently and effectively improving the model's knowledge without catastrophic forgetting. In this paper, we experiment on the corpus of code and math, yielding LLaMA Pro-8.3B, a versatile foundation model initialized from LLaMA2-7B, excelling in general tasks, programming, and mathematics. LLaMA Pro and its instruction-following counterpart (LLaMA Pro-Instruct) achieve advanced performance among various benchmarks, demonstrating superiority over existing open models in the LLaMA family and the immense potential of reasoning and addressing diverse tasks as an intelligent agent. Our findings provide valuable insights into integrating natural and programming languages, laying a solid foundation for developing advanced language agents that operate effectively in various environments.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论dweekly · 2024-01

“Paper (which is more useful than the linked HF model) is at https://arxiv.org/abs/2401.02415 There is also an -Instruct variant of this model published on HF. They are claiming the best balance of performance on language and coding tasks in an open source LLM of this size.”

LLaMA-Pro-8B原评论 ↗
评论sapphire42 · 2024-11

“They do actually make several claims as to the efficiency of the architecture compared to the Transformer, as you can see by the many graphs throughout the document. Their claim that their architecture is the only one that allows for gradually increasing the number of weights is a prominent one too, though, so I'll explain why I don't find that claim cred…”

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters原评论 ↗
被提交为 story · 2 次