“Summary from https://arxiv.org/pdf/2309.16609.pdf --- (q: how does one format lists on HN?) * qwen- {1.8B,7B,14B}: * 3 trillion tokens; start with BPE tiktoken, cl100k base vocab, augmented with chinese, numbers split into digits, final vocab 152k. * RoPE - rotary positional embedding * context length 2048 * qwen-14b perf percentages: 66.3 MMLU(5), 7…”
Qwen Technical Report
Large language models (LLMs) have revolutionized the field of artificial intelligence, enabling natural language processing tasks that were previously thought to be exclusive to humans. In this work, we introduce Qwen, the first installment of our large language model series. Qwen is a comprehensive language model series that encompasses distinct models with varying parameter counts. It includes Qwen, the base pretrained language models, and Qwen-Chat, the chat models finetuned with human alignment techniques. The base language models consistently demonstrate superior performance across a multitude of downstream tasks, and the chat models, particularly those trained using Reinforcement Learning from Human Feedback (RLHF), are highly competitive. The chat models possess advanced tool-use and planning capabilities for creating agent applications, showcasing impressive performance even when compared to bigger models on complex tasks like utilizing a code interpreter. Furthermore, we have developed coding-specialized models, Code-Qwen and Code-Qwen-Chat, as well as mathematics-focused models, Math-Qwen-Chat, which are built upon base language models. These models demonstrate significantly improved performance in comparison with open-source models, and slightly fall behind the proprietary models.
候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“>Meta calling their approach "open" and what the OSI deems "open" are both flawed and/or misleading.[...] It's gotten to the point that whenever I see a Western publication, I groan. llama weights are open, and Qwen 1.0 was based on llama,[0] so clearly those papers are useful to some people, even if some people want to be angry at Meta for not being ope…”
- Qwen OSS 7B,14B models; techreport - Code-Qwen, Math-Qwen [sep28'23] ↗saurabh20n · 2023-11-30 · 1 分