按年被提到
20232023 · 1 次2026
大模型与语言模型›混合专家与高效训练·arXiv:2309.03852 · cs.CL

FLM-101B: An Open LLM and How to Train It with $100K Budget

Xiang Li, Yiqun Yao, Xin Jiang, Xuezhi Fang, Xuying Meng, Siqi Fan, Peng Han, Jing Li, Li Du, Bowen Qin, Zheng Zhang, Aixin Sun 等 13 人 · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 2 人首次 2023-09-08 · 最近 2024-11-02
2不同的人
2活跃月份
2讨论串
15被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Large language models (LLMs) are considered important approaches towards foundational machine intelligence, achieving remarkable success in Natural Language Processing and multimodal tasks, among others. However, the carbon footprints and financial costs originating from heavy pre-training computation is a non-negligible issue. Progressive training methods, inspired by the neurogenesis process that grows neural structures, have shown potential to accelerate LLM pre-training. However, the algorithms, implementation, and practices for progressively training LLMs beyond 100B parameters remain underexplored. In this paper, we show that our model, namely FLM-101B, trained with our growth strategy under a budget of \$100K, reaches 80\% of the baselines' performances with only 10\% of their floating-point operations. We believe that further studies on progressive training will benefit the community by cutting down the costs and promoting green AI. The checkpoint of FLM-101B is released at https://huggingface.co/CofeAI/FLM-101B.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论sapphire42 · 2024-11

“They do actually make several claims as to the efficiency of the architecture compared to the Transformer, as you can see by the many graphs throughout the document. Their claim that their architecture is the only one that allows for gradually increasing the number of weights is a prominent one too, though, so I'll explain why I don't find that claim cred…”

TokenFormer: Rethinking Transformer Scaling with Tokenized Model Parameters原评论 ↗
被提交为 story · 1 次