mlsanityData to 7 Oct 2026

From Language Models to World-Acting Systems: Progress and Limits of Agentic AI across Digital, Social, Virtual, and Physical Environments

arXiv 2609.04894 ↗First posted 4 Sep 2026cs.AI cs.LG cs.MAcited by 1

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Large language models become consequential agents when surrounding systems let outputs change external state. Models now call tools, operate interfaces, delegate work, retain state, inhabit generated worlds, and control robots or laboratory equipment. Such advances are often narrated as one march toward autonomy, conflating model competence, system integration, persistence, and safe authority. This critical review synthesizes primary research and official technical specifications available by 31 August 2026. We organize the evidence along delegated authority, temporal persistence, and environmental coupling, while separating model, harness, and environment. Within the evidence examined, action-interface expansion is documented more convincingly than robust completion, recovery, authorization, or independent verification. Model Context Protocol and Agent2Agent improve interoperability but do not establish trustworthy delegation; multi-agent organization adds specialization alongside cost and correlated failure. Persistent simulations and world models support training and planning but do not themselves demonstrate agency; robotics and self-driving laboratories establish bounded feasibility rather than unattended open-world reliability. We propose justified delegation as an analytical and normative heuristic, not an observed law or certified score: expand action scope only where evidence supports provenance, bounded authority, failure detection, safe recovery, and calibrated human control. This framing yields a research agenda for coupled model-harness evaluation, capability-based permissions, durable state, cross-agent accountability, and staged physical validation.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 29 of its 51 references that are in the map. First posted on arXiv 4 Sep 2026.

Builds on

29 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026Single-Agent LLMs Outperform Multi-Agent Systems on Multi-Hop Reasoning Under Equal Thinking Token Budgets2 Apr
  2. 2025Magentic-UI: Towards Human-in-the-loop Agentic Systems30 Jul
  3. 2025Survey on Evaluation of LLM-based Agents20 Mar
  4. 2024Magentic-One: A Generalist Multi-Agent System for Solving Complex Tasks7 Nov
  5. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun
  6. 2023Cognitive Architectures for Language Agents5 Sep
  7. 2023AgentBench: Evaluating LLMs as Agents7 Aug

Other references · 22

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
  4. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,500
  5. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  6. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,081
  7. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  8. 2023AutoGen: Enabling Next-Gen LLM Applications via Multi-Agent Conversationcited by 1,906
  9. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  10. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  11. 2025Cosmos World Foundation Model Platform for Physical AIcited by 833
  12. 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
10 more
  1. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 609
  2. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
  3. 2024Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agentscited by 341
  4. 2026Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnessescited by 119
  5. 2024Scaling Instructable Agents Across Many Simulated Worldscited by 67
  6. 2025Authenticated Delegation and Authorized AI Agentscited by 63
  7. 2026Code as Agent Harnesscited by 40
  8. 2025Stay Focused: Problem Drift in Multi-Agent Debatecited by 23
  9. 2026VibeWorlding: Can Multimodal Agents Construct 3D Open Worlds End-to-End?cited by 2
  10. 2026Security Risks of AI Agents Hiring Humans: An Empirical Marketplace Studycited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Path Ahead for Agentic AI: Challenges and Opportunities
  2. 2026Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents
  3. 2026From Traces to Agentic Worlds: Agentic Language World Models for Interactive Environment Simulation
  4. 2023Evaluating Language-Model Agents on Realistic Autonomous Tasks
  5. 2026From Task Solving to Robust Real-World Adaptation in LLM Agents
  6. 2025A Survey on Agentic Multimodal Large Language Models
  7. 2026CoMAP: Co-Evolving World Models and Agent Policies for LLM Agents
  8. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forward