按年被提到
20242024 · 4 次2026
大模型与语言模型›大模型预训练与缩放·arXiv:2402.03175 · cs.LG

Beyond the Black Box: A Statistical Model for LLM Reasoning and Inference

Siddhartha Dalal, Vishal Misra · 2024
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 4 人首次 2024-04-28 · 最近 2024-05-08
4不同的人
2活跃月份
4讨论串
3被引(OpenAlex)
arXiv 摘要(候选层不写导读)

This paper introduces a novel Bayesian learning model to explain the behavior of Large Language Models (LLMs), focusing on their core optimization metric of next token prediction. We develop a theoretical framework based on an ideal generative text model represented by a multinomial transition probability matrix with a prior, and examine how LLMs approximate this matrix. Key contributions include: (i) a continuity theorem relating embeddings to multinomial distributions, (ii) a demonstration that LLM text generation aligns with Bayesian learning principles, (iii) an explanation for the emergence of in-context learning in larger models, (iv) empirical validation using visualizations of next token probabilities from an instrumented Llama model Our findings provide new insights into LLM functioning, offering a statistical foundation for understanding their capabilities and limitations. This framework has implications for LLM design, training, and application, potentially guiding future developments in the field.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论zipfcharge · 2024-05

“It's because an LLM is essentially a probability matrix. You type a prompt, then it calculates what's the probability of getting a next word and so on, eventually forming a sentence. The probability learned is based on the training data. Because of the underlying probability model, it's not going to be 100% deterministic. Plus a model like ChatGPT purpos…”

Consistency LLM: converting LLMs to parallel decoders accelerates inference 3.5x原评论 ↗
被提交为 story · 3 次