按年被提到
20232024 · 3 次2026
大模型与语言模型›长上下文与测试时计算·arXiv:2310.04444 · cs.CL

What's the Magic Word? A Control Theory of LLM Prompting

Aman Bhargava, Cameron Witkowski, Shi-Zhuo Looi, Matt Thomson · 2023
开始阅读 ↗PDFalphaXiv 讨论
被提及 · 3 人首次 2024-06-13 · 最近 2024-10-14
3不同的人
2活跃月份
3讨论串
20被引(OpenAlex)
arXiv 摘要(候选层不写导读)

Prompt engineering is crucial for deploying LLMs but is poorly understood mathematically. We formalize LLM systems as a class of discrete stochastic dynamical systems to explore prompt engineering through the lens of control theory. We offer a mathematical analysis of the limitations on the controllability of self-attention as a function of the singular values of the parameter matrices. We present complementary empirical results on the controllability of a panel of LLMs, including Falcon-7b, Llama-7b, and Falcon-40b. Given initial state $\mathbf x_0$ from Wikitext and prompts of length $k \leq 10$ tokens, we find that the "correct" next token is reachable at least 97% of the time, and that the top 75 most likely next tokens are reachable at least 85% of the time. Intriguingly, short prompt sequences can dramatically alter the likelihood of specific outputs, even making the least likely tokens become the most likely ones. This control-theoretic analysis of LLMs demonstrates the significant and poorly understood role of input sequences in steering output probabilities, offering a foundational perspective for enhancing language model system capabilities.

候选层:至少 2 个不同的人、在 2 个不同月份提到过,还没到主书架的 3 人 3 个月。没有做立场标注,所以只写“被提及”。

HN 书评

推荐在前,同类按时间 · 每篇最多保留 20 条,统计用全部提及
评论ValentinA23 · 2024-10

“>People really don't understand how much better LLM swarms get with more agents. I never hit a point of diminishing returns on text quality Could you elaborate please ? One use for swarms is to use multiple agents/prompts in place of one single agent with one long prompt in order to increase performance by splitting one big task into many. It is very ti…”

Swarm, a new agent framework by OpenAI原评论 ↗
被提交为 story · 2 次