“I'm biased (as an author), but I think Figure 3 is the key result, as well as Tables 4 and 5 in the appendix.”
Do LLMs write like humans? Variation in grammatical and rhetorical styles
用 Biber 语言学特征比较 LLM 与人类写作风格的差异
作者用 Llama 3 的多个变体和 GPT-4o,基于相同提示构造人类与 LLM 的平行语料,并用 Douglas Biber 的词汇、语法和修辞特征集进行比较。结果发现人类与 LLM、以及不同 LLM 之间存在系统性的风格差异,这些差异不因模型变大而消失,且指令微调模型的差异比基座模型更大。
评论者用它解释 LLM 文本为什么有可辨认的“AI 腔”:差异不只在用词,还在语法和修辞,且指令微调模型比基座模型更明显;作者本人也在评论中现身。
学习路径
读这篇之前,按顺序读书架上的 2 篇 · 参考文献来自 OpenAlex,顺序和要点由 Claude Code 排出候选里与语言模型写作风格直接相关的不多。这条路径只挑出 GPT-3(LLM 写作能力的起点)和一篇估计 LLM 参与文本比例的工作,作为理解研究动机的背景。
- 1
说明大模型仅凭提示就能写出流畅文本,是后续比较人类与 LLM 写作的起点。
读少样本提示与大规模语言模型的生成能力;不必细读各任务评测。
- 2
研究 AI 改写文本在真实语料中的占比,与目标论文区分人类与 LLM 文本的动机相关。
读如何用词汇分布估计 LLM 修改文本的比例,以及发现的用词偏好差异。
- 3Do LLMs write like humans? Variation in grammatical and rhetorical styles
目标论文:用 Biber 语言学特征比较 Llama 3 与 GPT-4o 和人类写作的语法与修辞风格。
读两套平行语料的构建、Biber 特征的统计方法,以及指令微调前后风格差异的结果。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“the authors of this paper[0] were able to use AI to figure out if a block of text was written by AI, so hopefully this becomes a tool available to educators very soon.”
“And the shift isn't just vocabulary, it's also in grammar and rhetorical style.”
“(See table 3 of the paper.)”
“Comparing base and instruction-tuned models, the base models are vaguely human in style, while instruction-tuned models systematically prefer certain types of grammar and style features.”
“Our experience is that LLMs like GPT-4o have a particular writing style”
看全部 11 条 →
“preprint at Do LLMs write like humans? Variation in grammatical and rhetorical styles. Working on extending this to more recent models and other grammatical features now”
“The main sources I used are these:”
“the writing style is not because of the training texts, but because of whatever tasks the LLM companies do in instruction tuning.”
“much more human-like.”
“can show specifically which writing features change: Do LLMs write like humans? Variation in grammatical and rhetorical styles”