mlsanityData to 7 Oct 2026

What's the Magic Word? A Control Theory of LLM Prompting

arXiv 2310.04444 ↗First posted 2 Oct 2023cs.CL cs.AI cs.LGcited by 41

Not on the map. Why?

Abstract (arXiv)

Prompt engineering is crucial for deploying LLMs but is poorly understood mathematically. We formalize LLM systems as a class of discrete stochastic dynamical systems to explore prompt engineering through the lens of control theory. We offer a mathematical analysis of the limitations on the controllability of self-attention as a function of the singular values of the parameter matrices. We present complementary empirical results on the controllability of a panel of LLMs, including Falcon-7b, Llama-7b, and Falcon-40b. Given initial state mathbf x0 from Wikitext and prompts of length k ≤ 10 tokens, we find that the "correct" next token is reachable at least 97% of the time, and that the top 75 most likely next tokens are reachable at least 85% of the time. Intriguingly, short prompt sequences can dramatically alter the likelihood of specific outputs, even making the least likely tokens become the most likely ones. This control-theoretic analysis of LLMs demonstrates the significant and poorly understood role of input sequences in steering output probabilities, offering a foundational perspective for enhancing language model system capabilities.

Key facts. Cited by 41 papers in the map (data through 7 Oct 2026). Builds on 37 of its 54 references that are in the map. First posted on arXiv 2 Oct 2023.

Builds on

37 of 54 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  6. 2014Explaining and Harnessing Adversarial Examplescited by 7,287
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  10. 2016Pointer Sentinel Mixture Modelscited by 2,992
  11. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
25 more
  1. 2022Emergent Abilities of Large Language Modelscited by 2,483
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2019Language Models as Knowledge Bases?cited by 1,922
  4. 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,302
  5. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  8. 2023Large Language Modelscited by 774
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2020Transformer Interpretability Beyond Attention Visualizationcited by 620
  11. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  12. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  13. 2022RLPrompt: Optimizing Discrete Text Prompts with Reinforcement Learningcited by 361
  14. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  15. 2015Towards AI-Complete Question Answering: A Set of Prerequisite Toy Taskscited by 309
  16. 2023Large Language Models as Tool Makerscited by 293
  17. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  18. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  19. 2022A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Modelscited by 128
  20. 2023Is ChatGPT a Good Sentiment Analyzer? A Preliminary Studycited by 96
  21. 2023Document-Level Machine Translation with Large Language Modelscited by 90
  22. 2022TEMPERA: Test-Time Prompting via Reinforcement Learningcited by 50
  23. 2022Toward Human Readable Prompt Tuning: Kubrick's The Shining is a good movie, and a good prompt too?cited by 49
  24. 2023Taming AI Bots: Controllability of Neural States in Large Language Modelscited by 22
  25. 2023Numeracy from Literacy: Data Science as an Emergent Skill from Large Language Modelscited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 14; 2025: 16; 2026: 10.

41 papers.

Sort:

By area:

  1. 2024Token-Budget-Aware LLM Reasoningcited by 242
  2. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  3. 2024Concise Thoughts: Impact of Output Length on LLM Reasoning and Costcited by 108
  4. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  5. 2025A Systematic Survey of Automatic Prompt Optimization Techniquescited by 61
  6. 2025Benchmarking Prompt Sensitivity in Large Language Modelscited by 38
  7. 2024Capabilities of Large Language Models in Control Engineering: A Benchmark Study on GPT-4, Claude 3 Opus, and Gemini 1.0 Ultracited by 36
  8. 2024SPRIG: Improving Large Language Model Performance by System Prompt Optimizationcited by 28
  9. 2024Evaluating the Prompt Steerability of Large Language Modelscited by 26
  10. 2025Drift No More? Context Equilibria in Multi-Turn LLM Interactionscited by 21
  11. 2024LiSeCo: Linear Semantic Control for Language Generationcited by 20
  12. 2023Prompt Engineering Through the Lens of Optimal Controlcited by 16
29 more
  1. 2025Recitation over Reasoning: How Cutting-Edge Language Models Can Fail on Elementary School-Level Reasoning Problems?cited by 15
  2. 2024CBF-LLM: Safe Control for LLM Alignmentcited by 8
  3. 2026Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Controlcited by 8
  4. 2024Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AIcited by 5
  5. 2024Prompt Bakingcited by 5
  6. 2025FanarGuard: A Culturally-Aware Moderation Filter for Arabic Language Modelscited by 5
  7. 2025Why is prompting hard? Understanding prompts on binary sequence predictorscited by 4
  8. 2024BWArea Model: Learning World Model, Inverse Dynamics, and Policy for Controllable Language Generationcited by 3
  9. 2026Steering Robustness into World Action Models via Mechanistic Interpretability and Optimal Controlcited by 3
  10. 2024Novel-WD: Exploring acquisition of Novel World Knowledge in LLMs Using Prefix-Tuningcited by 2
  11. 2025PAM: Training Policy-Aligned Moderation Filters at Scalecited by 2
  12. 2025From Long to Lean: Performance-aware and Adaptive Chain-of-Thought Compression via Multi-round Refinementcited by 2
  13. 2026Activation Steering of Video Generation Models via Reduced-Order Linear Optimal Controlcited by 2
  14. 2024Supertrust foundational alignment: mutual trust must replace permanent control for safe superintelligencecited by 1
  15. 2025CIE: Controlling Language Model Text Generations Using Continuous Signalscited by 1
  16. 2025Machine Learning for Detection and Analysis of Novel LLM Jailbreakscited by 1
  17. 2025ARISE: An Adaptive Resolution-Aware Metric for Test-Time Scaling Evaluation in Large Reasoning Modelscited by 1
  18. 2026Is my model "mind blurting"? Interpreting the dynamics of reasoning tokens with Recurrence Quantification Analysis (RQA)cited by 1
  19. 2026Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equationscited by 1
  20. 2026Training-Free Universal Approximation by Prompting Random Transformerscited by 1
  21. 2024Concept Based Continuous Prompts for Interpretable Text Classificationcited by 0
  22. 2025A Framework for Lightweight Responsible Prompting Recommendationcited by 0
  23. 2025Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Modelscited by 0
  24. 2025How Does Controllability Emerge In Language Models During Pretraining?cited by 0
  25. 2025Control Barrier Function for Aligning Large Language Modelscited by 0
  26. 2026PromptCD: Test-Time Behavior Enhancement via Polarity-Prompt Contrastive Decodingcited by 0
  27. 2026A geometric relation of the error introduced by sampling a language model's output distribution to its internal statecited by 0
  28. 2026Sustaining AI safety: Control-theoretic external impossibility, intrinsic necessity, and structural requirementscited by 0
  29. 2026On the Capability and Limitation of Hard Promptcited by 0