“It was already known that training on code improves performance on all reasoning tasks[1], so this is not really surprising.”
Language Models of Code are Few-Shot Commonsense Learners
把结构化常识推理改写成代码生成,Codex表现更好
论文把“生成事件图、推理图”这类结构化常识推理任务改写为代码生成任务,让用代码预训练的语言模型来完成。即使任务本身与代码无关,少样本设定下的 Codex 也优于在目标任务上微调的 T5 等自然语言模型,以及 GPT-3 等强基线。
HN 上的人常引它来支持“用代码训练的模型推理更强,即使基准与代码无关”这一说法,也有人认为这一点此前已知,并不意外。
学习路径
读这篇之前,按顺序读书架上的 3 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出这条路径帮我建立预训练语言模型、文本到文本范式以及用语言模型生成形式化代码的背景,再看把结构化常识生成改写成代码生成的做法。
- 1
- 2
文本到文本的统一框架,微调的 T5 一类序列化输出的做法正是目标论文所对照的。
读抓把任务都转成文本输入输出的格式,理解把图序列化成文本的做法。
- 3
- 4Language Models of Code are Few-Shot Commonsense Learners
把结构化常识推理的输出图改写成代码,让代码语言模型少样本完成。
读抓图到代码的转换方式、提示的设计,以及与序列化为扁平文本的对比实验。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“Alternatively, as Madaan et al. show in their 2022 paper[2],”
“LLMs trained on code reason better. Perform better on reasoning benchmarks even if the benchmarks have nothing to do with code.”
“Language models trained on Code reason better, even on benchmarks that have nothing to do with code.”
“Language models of code reason better even if the benchmarks have nothing to do with code.”
- Code trained LLMs reason better, on benchmarks that have nothing to do with code ↗og_kalu · 2023-05-13 · 2 分
- Language Models of Code Are Few-Shot Commonsense Learners ↗dnqthao · 2023-05-13 · 1 分
- Language Models of Code Are Few-Shot Commonsense Learners ↗tim_sw · 2023-05-15 · 1 分