The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
Foundation model agents are increasingly deployed for real-world decision-making, but suffer from the sim-to-real gap. While robotics and classical control have mature frameworks to address this gap, the foundation model community is treating agent robustness as an entirely novel phenomenon. Our paper proposes formalizing the foundation model agent evaluation and training gap as a classical sim-to-real problem structured entirely around the four elements of a Markov Decision Process, including Observation, Action, Transition, and Reward. In this paper, we set a comprehensive research agenda that translates classical discrepancies into the foundation model domain and advocates for adopting established solutions like domain randomization. We provide concrete examples, such as a multilingual tool calling to demonstrate how severe observation space gaps lead to operationally invalid actions despite correct semantic intent. Ultimately, this agenda aims to drive a paradigm shift, yielding a unified vocabulary and standardized stress test benchmarks to foster a new generation of highly trustworthy agents for reliable real-world applications.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 35 of its 55 references that are in the map. First posted on arXiv 5 Jun 2026.
Builds on
35 of 55 references.
Closest ancestors · 4
Same direction, newest first
- 2026AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition11 Feb
- 2025The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective4 Jun
- 2024Understanding the planning of LLM agents: A survey5 Feb
- 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society31 Mar
Other references · 31
- 2017Safe Reinforcement Learning via Shieldingcited by 449
- 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 413
- 2019Active Domain Randomizationcited by 170
- 2025ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoningcited by 151
- 2025Where LLM Agents Fail and How They can Learn From Failurescited by 133
- 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
- 2025VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Usecited by 96
- 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 95
- 2023T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Stepcited by 93
- 2015Increasing the Action Gap: New Operators for Reinforcement Learningcited by 49
- 2025A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Modelscited by 38
- 2023Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learningcited by 36
19 more
- 2020Robustifying Reinforcement Learning Agents via Action Space Adversarial Trainingcited by 30
- 2018At Human Speed: Deep Reinforcement Learning with Action Delaycited by 26
- 2024LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentationcited by 23
- 2020Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamicscited by 21
- 2025On the Robustness of Agentic Function Callingcited by 19
- 2020Reinforced Grounded Action Transformation for Sim-to-Real Transfercited by 18
- 2017Reinforcement Learning for Pivoting Taskcited by 17
- 2024Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitationcited by 17
- 2020Stochastic Grounded Action Transformation for Robot Learning in Simulationcited by 16
- 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
- 2026Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Modelscited by 10
- 2024Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Spacecited by 9
- 2025Evaluating Robustness of Large Language Models Against Multilingual Typographical Errorscited by 9
- 2019A new Potential-Based Reward Shaping for Reinforcement Learning Agentcited by 8
- 2024Action Gaps and Advantages in Continuous-Time Distributional Reinforcement Learningcited by 7
- 2026When Simulation Lies: A Sim-to-Real Benchmark and Domain-Randomized RL Recipe for Tool-Use Agentscited by 6
- 2026Beyond Perfect APIs: A Comprehensive Evaluation of LLM Agents Under Real-World API Complexitycited by 5
- 2024Safety-Driven Deep Reinforcement Learning Framework for Cobots: A Sim2Real Approachcited by 3
- 2025Revealing the Challenges of Sim-to-Real Transfer in Model-Based Reinforcement Learning via Latent Space Modelingcited by 1
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2025Foundation Model Driven Robotics: A Comprehensive Review
- 2026Towards Agentic Robots: Model, Data, Environment, and Harness
- 2026Foundation Models in Robotics: A Comprehensive Review of Methods, Models, Datasets, Challenges and Future Research Directions
- 2024R-AIF: Solving Sparse-Reward Robotic Tasks from Pixels with Active Inference and World Models
- 2025Foundation Models as World Models: A Foundational Study in Text-Based GridWorlds
- 2024A Taxonomy of Architecture Options for Foundation Model-based Agents: Analysis and Decision Model
- 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Own
- 2025The Missing Reward: Active Inference in the Era of Experience