“There has also been some academic work linking the two, like this[3] paper.”
Large Language Models as Markov Chains
证明自回归 Transformer 语言模型等价于有限状态空间上的马尔可夫链
论文针对 LLM 泛化能力缺乏全面理论分析的现状,建立了基于 Transformer 的自回归语言模型与有限状态空间上马尔可夫链的等价关系,从而可以从第一性原理研究 LLM 的多步推理机制。作者把所得结果与 LLM 的病态行为(如重复和高温度下的不连贯回复)联系起来,并在现实的数据和模型假设下推导出预训练和上下文学习的泛化界。在最新的 Llama 和 Gemma 系列模型上的实验表明,理论能够正确刻画它们在实践中的行为。
HN 上的人在争论 LLM 是不是“只是马尔可夫链”时反复引用它,作为二者可以被形式化地等价、并能用马尔可夫理论分析 LLM 的依据。
HN 书评
推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及“we covered this paper in one of our internal journal clubs a few weeks ago”
“I did find this paper on the topic[0], but I need to give it a better read.”
“And yes, LLM can be studied under the lens of Markov processes: Large Language Models as Markov Chains”
“If you want to steal code, you can take it from GitHub and strip the license. That is what the Markov chains do.”
“In fact, you can use Markov Chain theory as an analytical tool to study LLMs: Large Language Models as Markov Chains”
看全部 9 条 →
“But of course there also is the equivalence between LLMs and Markov chains.”
“Large Language Models as Markov Chains describes the equivalence formally.”
“"Large Language Models as Markov Chains"”
- Large Language Models as Markov Chains ↗tosh · 2024-10-08 · 1 分
- LLMs as Markov Chains ↗akrymski · 2024-10-08 · 5 分
- Large Language Models as Markov Chains ↗Anon84 · 2024-11-30 · 75 分