Interpretable Risk Mitigation in LLM Agent Systems
On the map: LLM agents and tool use → Social simulation and multi-agent debate
Abstract (arXiv)
Autonomous agents powered by large language models (LLMs) enable novel use cases in domains where responsible action is increasingly important. Yet the inherent unpredictability of LLMs raises safety concerns about agent reliability. In this work, we explore agent behaviour in a toy, game-theoretic environment based on a variation of the Iterated Prisoner's Dilemma. We introduce a strategy-modification method-independent of both the game and the prompt-by steering the residual stream with interpretable features extracted from a sparse autoencoder latent space. Steering with the good-faith negotiation feature lowers the average defection probability by 28 percentage points. We also identify feasible steering ranges for several open-source LLM agents. Finally, we hypothesise that game-theoretic evaluation of LLM agents, combined with representation-steering alignment, can generalise to real-world applications on end-user devices and embodied platforms.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Social simulation and multi-agent debate, one of 16 directions in LLM agents and tool use. Builds on 41 of its 68 references that are in the map. First posted on arXiv 15 May 2025.
Builds on
41 of 68 references.
Closest ancestors · 1
Same direction, newest first
Other references · 40
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2021Evaluating Large Language Models Trained on Codecited by 8,320
- 2020Measuring Massive Multitask Language Understandingcited by 7,963
- 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
- 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
- 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
- 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
- 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
28 more
- 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
- 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
- 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,061
- 2016Concrete Problems in AI Safetycited by 1,642
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
- 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
- 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
- 2022Toy Models of Superpositioncited by 971
- 2020A Primer in BERTology: What we know about how BERT workscited by 815
- 2016The Mythos of Model Interpretabilitycited by 721
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
- 2022Towards Reasoning in Large Language Models: A Surveycited by 596
- 2020Artificial Intelligence, Values and Alignmentcited by 448
- 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
- 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 293
- 2023Playing repeated games with Large Language Modelscited by 217
- 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
- 2016Research Priorities for Robust and Beneficial Artificial Intelligencecited by 74
- 2024Nicer Than Humans: How do Large Language Models Behave in the Prisoner's Dilemma?cited by 63
- 2024Moral Alignment for LLM Agentscited by 39
- 2024Large Language Models for UAVs: Current State and Pathways to the Futurecited by 33
- 2023An evolutionary model of personality traits related to cooperative behavior using a large language modelcited by 18
- 2024Non-Linear Inference Time Intervention: Improving LLM Truthfulnesscited by 9
- 2024Large Model Strategic Thinking, Small Model Efficiency: Transferring Theory of Mind in Large Language Modelscited by 7
- 2024Integrated Design and Governance of Agentic AI Systems through Adaptive Information Modulationcited by 4
Built on by
Citing papers per year: 2025: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2024Practical Considerations for Agentic LLM Systems
- 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
- 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspective
- 2024STRIDE: A Tool-Assisted LLM Agent Framework for Strategic and Interactive Decision-Making
- 2024WiS Platform: Enhancing Evaluation of LLM-Based Multi-Agent Systems Through Game-Based Analysis
- 2025Understanding LLM Agent Behaviours via Game Theory: Strategy Recognition, Biases and Multi-Agent Dynamics
- 2025LLMs Working in Harmony: A Survey on the Technological Aspects of Building Effective LLM-Based Multi Agent Systems
- 2026Game-Theoretic Lens on LLM-based Multi-Agent Systems