mlsanityData to 14 Oct 2026

Infrastructure for AI Agents

arXiv 2501.10114 ↗First posted 17 Jan 2025cs.AIcited by 41

Not on the map. Why?

Abstract (arXiv)

AI agents plan and execute interactions in open-ended environments. For example, OpenAI's Operator can use a web browser to do product comparisons and buy online goods. Much research on making agents useful and safe focuses on directly modifying their behaviour, such as by training them to follow user instructions. Direct behavioural modifications are useful, but do not fully address how heterogeneous agents will interact with each other and other actors. Rather, we will need external protocols and systems to shape such interactions. For instance, agents will need more efficient protocols to communicate with each other and form agreements. Attributing an agent's actions to a particular human or other legal entity can help to establish trust, and also disincentivize misuse. Given this motivation, we propose the concept of agent infrastructure: technical systems and shared protocols external to agents that are designed to mediate and influence their interactions with and impacts on their environments. Just as the Internet relies on protocols like HTTPS, our work argues that agent infrastructure will be similarly indispensable to ecosystems of agents. We identify three functions for agent infrastructure: 1) attributing actions, properties, and other information to specific agents, their users, or other actors; 2) shaping agents' interactions; and 3) detecting and remedying harmful actions from agents. We provide an incomplete catalog of research directions for such functions. For each direction, we include analysis of use cases, infrastructure adoption, relationships to existing (internet) infrastructure, limitations, and open questions. Making progress on agent infrastructure can prepare society for the adoption of more advanced agents.

Key facts. Cited by 41 papers in the map (data through 14 Oct 2026). Builds on 58 of its 137 references that are in the map. First posted on arXiv 17 Jan 2025.

Builds on

58 of 137 references.

References · 58

  1. 2017Deep reinforcement learning from human preferencescited by 3,912
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
  4. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
  6. 2023GAIA: a benchmark for General AI Assistantscited by 1,128
  7. 2024The Instruction Hierarchy: Training LLMs to Prioritize Privileged Instructionscited by 493
  8. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  9. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  11. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  12. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 303
38 more
  1. 2016Cooperative Inverse Reinforcement Learningcited by 300
  2. 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
  3. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 229
  4. 2024AI Agents That Mattercited by 219
  5. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  6. 2024RE-Bench: Evaluating frontier AI R&D capabilities of language model agents against human expertscited by 175
  7. 2024Agent Smith: A Single Image Can Jailbreak One Million Multimodal LLM Agents Exponentially Fastcited by 153
  8. 2023Large Language Models Understand and Can be Enhanced by Emotional Stimulicited by 146
  9. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 138
  10. 2024Tamper-Resistant Safeguards for Open-Weight LLMscited by 138
  11. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 135
  12. 2024LLM Agents can Autonomously Exploit One-day Vulnerabilitiescited by 128
  13. 2024Collective Constitutional AI: Aligning a Language Model with Public Inputcited by 123
  14. 2024The Ethics of Advanced AI Assistantscited by 107
  15. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  16. 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
  17. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  18. 2024Visibility into AI Agentscited by 79
  19. 2024Beyond Browsing: API-Based Web Agentscited by 78
  20. 2022Outsider Oversight: Designing a Third Party Audit Ecosystem for AI Governancecited by 67
  21. 2024Algorithmic Collusion by Large Language Modelscited by 62
  22. 2024Here Comes The AI Worm: Unleashing Zero-click Worms that Target GenAI-Powered Applicationscited by 59
  23. 2023Who Audits the Auditors? Recommendations from a field scan of the algorithmic auditing ecosystemcited by 52
  24. 2024Learning From Failure: Integrating Negative Examples when Fine-tuning Large Language Models as Agentscited by 50
  25. 2024A Scalable Communication Protocol for Networks of Large Language Modelscited by 45
  26. 2021The Flaws of Policies Requiring Human Oversight of Government Algorithmscited by 44
  27. 2024Adversaries Can Misuse Combinations of Safe Modelscited by 32
  28. 2023Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Webcited by 24
  29. 2024GoEX: Perspectives and Designs Towards a Runtime for Autonomous LLM Applicationscited by 23
  30. 2019The Role of Cooperation in Responsible AI Developmentcited by 21
  31. 2024IDs for AI Systemscited by 19
  32. 2024Personhood credentials: Artificial intelligence and the value of privacy-preserving tools to distinguish who is real onlinecited by 16
  33. 2023An Empirical Study & Evaluation of Modern CAPTCHAscited by 13
  34. 2024Scenarios for the Transition to AGIcited by 13
  35. 2023WebWISE: Web Interface Control and Sequential Exploration with Large Language Modelscited by 11
  36. 2024Societal Adaptation to Advanced AIcited by 10
  37. 2021An Object Detection based Solver for Google's Image reCAPTCHA v2cited by 8
  38. 2023Designing Fiduciary Artificial Intelligencecited by 8

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 22; 2026: 19.

41 papers.

Sort:

By area:

  1. 2025OpenAgentSafety: A Comprehensive Framework for Evaluating Real-World AI Agent Safetycited by 75
  2. 2025Open Challenges in Multi-Agent Security: Towards Secure Systems of Interacting AI Agentscited by 71
  3. 2025SAGA: A Security Architecture for Governing AI Agentic Systemscited by 45
  4. 2025Reliable Weak-to-Strong Monitoring of LLM Agentscited by 31
  5. 2026The 2025 AI Agent Index: Documenting Technical and Safety Features of Deployed Agentic AI Systemscited by 30
  6. 2025Virtual Agent Economiescited by 21
  7. 2025Risk Analysis Techniques for Governed LLM-based Multi-Agent Systemscited by 18
  8. 2025An Economy of AI Agentscited by 18
  9. 2025Inherent and emergent liability issues in LLM-based agentic systems: a principal-agent perspectivecited by 13
  10. 2025Distributional AGI Safetycited by 12
  11. 2025EconEvals: Benchmarks and Litmus Tests for Economic Decision-Making by LLM Agentscited by 10
  12. 2025The Singapore Consensus on Global AI Safety Research Prioritiescited by 10
29 more
  1. 2025A Call for Collaborative Intelligence: Why Human-Agent Systems Should Precede AI Autonomycited by 9
  2. 2025Resource Rational Contractualism Should Guide AI Alignmentcited by 8
  3. 2026Right to History: A Sovereignty Kernel for Verifiable AI Agent Executioncited by 7
  4. 2025Position: Stop Acting Like Language Model Agents Are Normal Agentscited by 5
  5. 2026A Case Study on Emergent Cheating and Whistleblowing in Autonomous Research Swarmscited by 5
  6. 2025Typed Chain-of-Thought: A Curry-Howard Framework for Verifying LLM Reasoningcited by 3
  7. 2025Agentic Inequalitycited by 3
  8. 2025On the Regulatory Potential of User Interfaces for AI Agent Governancecited by 3
  9. 2026Architecting Trust in Artificial Epistemic Agentscited by 3
  10. 2025Against racing to AGI: Cooperation, deterrence, and catastrophic riskscited by 2
  11. 2026Legal Infrastructure for Transformative AI Governancecited by 2
  12. 2026Attacks and Mitigations for Distributed Governance of Agentic AI under Byzantine Adversariescited by 2
  13. 2026Steganalysis of Adaptive Covert Collusion in Tool-Using Agent Populations: A Black-Box, Cross-Principal Approachcited by 2
  14. 2025From Turing to Tomorrow: The UK's Approach to AI Regulationcited by 1
  15. 2025Dimensional Characterization and Pathway Modeling for Catastrophic AI Riskscited by 1
  16. 2025Operationalising Extended Cognition: Formal Metrics for Corporate Knowledge and Legal Accountabilitycited by 1
  17. 2026ClawNet: Human-Symbiotic Agent Network for Cross-User Autonomous Cooperationcited by 1
  18. 2026Comprehensive AI governance requires addressing non-model gainscited by 1
  19. 2026A Note on the Strategic Confinement Problemcited by 1
  20. 2026Infrastructure for the Agentic Web: Gap Analysis and Architecture from the Agentverse Platformcited by 1
  21. 2025Towards Measurement Theory for Artificial Intelligencecited by 0
  22. 2026Governance by Design: A Parsonian Institutional Architecture for Internet-Wide Agent Societiescited by 0
  23. 2026Precautionary Governance of Autonomous AI: Legal Personhood as Functional Instrumentcited by 0
  24. 2026Governed Individuation: Cryptographically Decoupling an Agent's Learning from Its Authoritycited by 0
  25. 2026Binding Biometrics with AI Agent Identifiers for Delegation of Authoritycited by 0
  26. 2026PrimeAgentOrchestrator: Memory-Primed Agent Spawning for Personal AI Infrastructurecited by 0
  27. 2026Can escalation channels redirect reward hacking toward defect disclosure?cited by 0
  28. 2026The Civilization Framework: Sovereign-Anchored Communication Between Personal Multi-Agent Systemscited by 0
  29. 2026When the Agent Becomes the Kernel: A Systematization of Security on the Path to AI-Native Operating Systemscited by 0