# Regulating the Agency of LLM-based Agents

arXiv: https://arxiv.org/abs/2509.22735 · HTML: https://mlsanity.com/paper/2509.22735/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 23 of its 51 references that are in the map.
- First posted on arXiv 25 Sep 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 23 of 23)

- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [AgentBench: Evaluating LLMs as Agents](https://mlsanity.com/paper/2308.03688/) (2023)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [Agent-SafetyBench: Evaluating the Safety of LLM Agents](https://mlsanity.com/paper/2412.14470/) (2024)
- [Frontier Models are Capable of In-context Scheming](https://mlsanity.com/paper/2412.04984/) (2024)
- [Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMs](https://mlsanity.com/paper/2502.17424/) (2025)
- [Improving Instruction-Following in Language Models through Activation Steering](https://mlsanity.com/paper/2410.12877/) (2024)
- [Measuring AI Ability to Complete Long Software Tasks](https://mlsanity.com/paper/2503.14499/) (2025)
- [AI Sandbagging: Language Models can Strategically Underperform on Evaluations](https://mlsanity.com/paper/2406.07358/) (2024)
- [Harms from Increasingly Agentic Algorithmic Systems](https://mlsanity.com/paper/2302.10329/) (2023)
- [Evaluating Language-Model Agents on Realistic Autonomous Tasks](https://mlsanity.com/paper/2312.11671/) (2023)
- [Subliminal Learning: Language models transmit behavioral traits via hidden signals in data](https://mlsanity.com/paper/2507.14805/) (2025)
- [WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Tasks](https://mlsanity.com/paper/2407.05291/) (2024)
- [Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents](https://mlsanity.com/paper/2502.15840/) (2025)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?](https://mlsanity.com/paper/2502.15657/) (2025)
- [Thought Crime: Backdoors and Emergent Misalignment in Reasoning Models](https://mlsanity.com/paper/2506.13206/) (2025)
- [Levels of Autonomy for AI Agents](https://mlsanity.com/paper/2506.12469/) (2025)
- [Designing a Dashboard for Transparency and Control of Conversational AI](https://mlsanity.com/paper/2406.07882/) (2024)
- [Measuring AI agent autonomy: Towards a scalable approach with code inspection](https://mlsanity.com/paper/2502.15212/) (2025)
- [An international treaty to implement a global compute cap for advanced artificial intelligence](https://mlsanity.com/paper/2311.10748/) (2023)
- [Toward a Global Regime for Compute Governance: Building the Pause Button](https://mlsanity.com/paper/2506.20530/) (2025)

## Built on by (most cited first, 1 of 1)

- [A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework](https://mlsanity.com/paper/2604.23338/) (2026)

## Abstract (arXiv)

As increasingly capable large language model (LLM)-based agents are developed, the potential harms caused by misalignment and loss of control grow correspondingly severe. To address these risks, we propose an approach that directly measures and controls the agency of these AI systems. We conceptualize the agency of LLM-based agents as a property independent of intelligence-related measures and consistent with the interdisciplinary literature on the concept of agency. We offer (1) agency as a system property operationalized along the dimensions of preference rigidity, independent operation, and goal persistence, (2) a representation engineering approach to the measurement and control of the agency of an LLM-based agent, and (3) regulatory tools enabled by this approach: mandated testing protocols, domain-specific agency limits, insurance frameworks that price risk based on agency, and agency ceilings to prevent societal-scale risks. We view our approach as a step toward reducing the risks that motivate the ``Scientist AI'' paradigm, while still capturing some of the benefits from limited agentic behavior.