mlsanityData to 14 Oct 2026

Regulating the Agency of LLM-based Agents

arXiv 2509.22735 ↗First posted 25 Sep 2025cs.CY cs.AIcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

As increasingly capable large language model (LLM)-based agents are developed, the potential harms caused by misalignment and loss of control grow correspondingly severe. To address these risks, we propose an approach that directly measures and controls the agency of these AI systems. We conceptualize the agency of LLM-based agents as a property independent of intelligence-related measures and consistent with the interdisciplinary literature on the concept of agency. We offer (1) agency as a system property operationalized along the dimensions of preference rigidity, independent operation, and goal persistence, (2) a representation engineering approach to the measurement and control of the agency of an LLM-based agent, and (3) regulatory tools enabled by this approach: mandated testing protocols, domain-specific agency limits, insurance frameworks that price risk based on agency, and agency ceilings to prevent societal-scale risks. We view our approach as a step toward reducing the risks that motivate the ``Scientist AI'' paradigm, while still capturing some of the benefits from limited agentic behavior.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 23 of its 51 references that are in the map. First posted on arXiv 25 Sep 2025.

Builds on

23 of 51 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025Vending-Bench: A Benchmark for Long-Term Coherence of Autonomous Agents20 Feb
  2. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  4. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks18 Dec
  5. 2023AgentBench: Evaluating LLMs as Agents7 Aug
  6. 2023Harms from Increasingly Agentic Algorithmic Systems20 Feb

Other references · 17

  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  2. 2024Agent-SafetyBench: Evaluating the Safety of LLM Agentscited by 306
  3. 2024Frontier Models are Capable of In-context Schemingcited by 287
  4. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 172
  5. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 160
  6. 2025Measuring AI Ability to Complete Long Software Taskscited by 156
  7. 2024AI Sandbagging: Language Models can Strategically Underperform on Evaluationscited by 145
  8. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 95
  9. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 90
  10. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  11. 2025Superintelligent Agents Pose Catastrophic Risks: Can Scientist AI Offer a Safer Path?cited by 75
  12. 2025Thought Crime: Backdoors and Emergent Misalignment in Reasoning Modelscited by 65
5 more
  1. 2025Levels of Autonomy for AI Agentscited by 48
  2. 2024Designing a Dashboard for Transparency and Control of Conversational AIcited by 46
  3. 2025Measuring AI agent autonomy: Towards a scalable approach with code inspectioncited by 10
  4. 2023An international treaty to implement a global compute cap for advanced artificial intelligencecited by 4
  5. 2025Toward a Global Regime for Compute Governance: Building the Pause Buttoncited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Frameworkcited by 11

Similar, not cited

Close in wording, no citation either way

  1. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Survey
  3. 2025Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
  4. 2025Is the `Agent' Paradigm a Limiting Framework for Next-Generation Intelligent Systems?
  5. 2025Reliable agent engineering should integrate machine-compatible organizational principles
  6. 2025Do LLMs trust AI regulation? Emerging behaviour of game-theoretic LLM agents
  7. 2026Organizational Control Layer: Governance Infrastructure at the Execution Boundary of LLM Agent Systems
  8. 2023Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Models