# Taming AI Bots: Controllability of Neural States in Large Language Models

arXiv: https://arxiv.org/abs/2305.18449 · HTML: https://mlsanity.com/paper/2305.18449/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 29 May 2023.

## Builds on (most cited first, 22 of 22)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Fine-Tuning Language Models from Human Preferences](https://mlsanity.com/paper/1909.08593/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [WebGPT: Browser-assisted question-answering with human feedback](https://mlsanity.com/paper/2112.09332/) (2021)
- [Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?](https://mlsanity.com/paper/2202.12837/) (2022)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [What Can Transformers Learn In-Context? A Case Study of Simple Function Classes](https://mlsanity.com/paper/2208.01066/) (2022)
- [What learning algorithm is in-context learning? Investigations with linear models](https://mlsanity.com/paper/2211.15661/) (2022)
- [Larger language models do in-context learning differently](https://mlsanity.com/paper/2303.03846/) (2023)
- [Pretraining Language Models with Human Preferences](https://mlsanity.com/paper/2302.08582/) (2023)
- [Emergence of Invariance and Disentanglement in Deep Representations](https://mlsanity.com/paper/1706.01350/) (2017)
- [A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models](https://mlsanity.com/paper/2201.05337/) (2022)
- [Where is the Information in a Deep Neural Network?](https://mlsanity.com/paper/1905.12213/) (2019)
- [LQF: Linear Quadratic Fine-Tuning](https://mlsanity.com/paper/2012.11140/) (2020)
- [Estimating informativeness of samples with Smooth Unique Information](https://mlsanity.com/paper/2101.06640/) (2021)
- [On the Learnability of Physical Concepts: Can a Neural Network Understand What's Real?](https://mlsanity.com/paper/2207.12186/) (2022)

## Built on by (most cited first, 22 of 22)

- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [What's the Magic Word? A Control Theory of LLM Prompting](https://mlsanity.com/paper/2310.04444/) (2023)
- [Cycles of Thought: Measuring LLM Confidence through Stable Explanations](https://mlsanity.com/paper/2406.03441/) (2024)
- [LiSeCo: Linear Semantic Control for Language Generation](https://mlsanity.com/paper/2405.15454/) (2024)
- [Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attacks](https://mlsanity.com/paper/2503.00187/) (2025)
- [Meaning Representations from Trajectories in Autoregressive Models](https://mlsanity.com/paper/2310.18348/) (2023)
- [Activation Steering with a Feedback Controller](https://mlsanity.com/paper/2510.04309/) (2025)
- [Preemptive Detection and Steering of LLM Misalignment via Latent Reachability](https://mlsanity.com/paper/2509.21528/) (2025)
- [CBF-LLM: Safe Control for LLM Alignment](https://mlsanity.com/paper/2408.15625/) (2024)
- [Local Linearity of LLMs Enables Activation Steering via Model-Based Linear Optimal Control](https://mlsanity.com/paper/2604.19018/) (2026)
- [Meanings and Feelings of Large Language Models: Observability of Latent States in Generative AI](https://mlsanity.com/paper/2405.14061/) (2024)
- [Multi-Objective Linguistic Control of Large Language Models](https://mlsanity.com/paper/2406.16229/) (2024)
- [Towards Visual Foundational Models of Physical Scenes](https://mlsanity.com/paper/2306.03727/) (2023)
- [Can I say, now machines can think?](https://mlsanity.com/paper/2307.07526/) (2023)
- [State space models, emergence, and ergodicity: How many parameters are needed for stable predictions?](https://mlsanity.com/paper/2409.13421/) (2024)
- [GenCtrl -- A Formal Controllability Toolkit for Generative Models](https://mlsanity.com/paper/2601.05637/) (2026)
- [Understanding the Theoretical Foundations of Deep Neural Networks through Differential Equations](https://mlsanity.com/paper/2603.18331/) (2026)
- [Conjuring Semantic Similarity](https://mlsanity.com/paper/2410.16431/) (2024)
- [Dual Filter: A Transformer-like Inference Architecture for Hidden Markov Models](https://mlsanity.com/paper/2505.00818/) (2025)
- [Control Barrier Function for Aligning Large Language Models](https://mlsanity.com/paper/2511.03121/) (2025)
- [As Language Models Scale, Low-order Linear Depth Dynamics Emerge](https://mlsanity.com/paper/2603.12541/) (2026)
- [How does Chain of Thought decompose complex tasks?](https://mlsanity.com/paper/2604.08872/) (2026)

## Abstract (arXiv)

We tackle the question of whether an agent can, by suitable choice of prompts, control an AI bot to any state. To that end, we first introduce a formal definition of ``meaning'' that is amenable to analysis. Then, we characterize ``meaningful data'' on which large language models (LLMs) are ostensibly trained, and ``well-trained LLMs'' through conditions that are largely met by today's LLMs. While a well-trained LLM constructs an embedding space of meanings that is Euclidean, meanings themselves do not form a vector (linear) subspace, but rather a quotient space within. We then characterize the subset of meanings that can be reached by the state of the LLMs for some input prompt, and show that a well-trained bot can reach any meaning albeit with small probability. We then introduce a stronger notion of controllability as em almost certain reachability, and show that, when restricted to the space of meanings, an AI bot is controllable. We do so after introducing a functional characterization of attentive AI bots, and finally derive necessary and sufficient conditions for controllability. The fact that AI bots are controllable means that an adversary could steer them towards any state. However, the sampling process can be designed to counteract adverse actions and avoid reaching undesirable regions of state space before their boundary is crossed.