# The Sim-to-Real Gap of Foundation Model Agents: A Unified MDP Perspective

arXiv: https://arxiv.org/abs/2606.07017 · HTML: https://mlsanity.com/paper/2606.07017/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 5 Jun 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 35)

- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [Safe Reinforcement Learning via Shielding](https://mlsanity.com/paper/1708.08611/) (2017)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [ShieldAgent: Shielding Agents via Verifiable Safety Policy Reasoning](https://mlsanity.com/paper/2503.22738/) (2025)
- [Where LLM Agents Fail and How They can Learn From Failures](https://mlsanity.com/paper/2509.25370/) (2025)
- [Granite Code Models: A Family of Open Foundation Models for Code Intelligence](https://mlsanity.com/paper/2405.04324/) (2024)
- [VerlTool: Towards Holistic Agentic Reinforcement Learning with Tool Use](https://mlsanity.com/paper/2509.01055/) (2025)
- [Uncertainty Quantification and Confidence Calibration in Large Language Models: A Survey](https://mlsanity.com/paper/2503.15850/) (2025)
- [T-Eval: Evaluating the Tool Utilization Capability of Large Language Models Step by Step](https://mlsanity.com/paper/2312.14033/) (2023)
- [Increasing the Action Gap: New Operators for Reinforcement Learning](https://mlsanity.com/paper/1512.04860/) (2015)
- [The Cost of Dynamic Reasoning: Demystifying AI Agents and Test-Time Scaling from an AI Infrastructure Perspective](https://mlsanity.com/paper/2506.04301/) (2025)
- [A Survey of Sim-to-Real Methods in RL: Progress, Prospects and Challenges with Foundation Models](https://mlsanity.com/paper/2502.13187/) (2025)
- [Prompt to Transfer: Sim-to-Real Transfer for Traffic Signal Control with Prompt Learning](https://mlsanity.com/paper/2308.14284/) (2023)
- [Robustifying Reinforcement Learning Agents via Action Space Adversarial Training](https://mlsanity.com/paper/2007.07176/) (2020)
- [At Human Speed: Deep Reinforcement Learning with Action Delay](https://mlsanity.com/paper/1810.07286/) (2018)
- [LLM Uncertainty Quantification through Directional Entailment Graph and Claim Level Response Augmentation](https://mlsanity.com/paper/2407.00994/) (2024)
- [AgentNoiseBench: Benchmarking Robustness of Tool-Using LLM Agents Under Noisy Condition](https://mlsanity.com/paper/2602.11348/) (2026)
- [Crossing The Gap: A Deep Dive into Zero-Shot Sim-to-Real Transfer for Dynamics](https://mlsanity.com/paper/2008.06686/) (2020)
- [On the Robustness of Agentic Function Calling](https://mlsanity.com/paper/2504.00914/) (2025)
- [Reinforced Grounded Action Transformation for Sim-to-Real Transfer](https://mlsanity.com/paper/2008.01279/) (2020)
- [Reinforcement Learning for Pivoting Task](https://mlsanity.com/paper/1703.00472/) (2017)
- [Off-Dynamics Reinforcement Learning via Domain Adaptation and Reward Augmented Imitation](https://mlsanity.com/paper/2411.09891/) (2024)
- [Stochastic Grounded Action Transformation for Robot Learning in Simulation](https://mlsanity.com/paper/2008.01281/) (2020)
- [Mind the Gap: Offline Policy Optimization for Imperfect Rewards](https://mlsanity.com/paper/2302.01667/) (2023)
- [Lost in Execution: On the Multilingual Robustness of Tool Calling in Large Language Models](https://mlsanity.com/paper/2601.05366/) (2026)
- [Robust Deep Reinforcement Learning with Adaptive Adversarial Perturbations in Action Space](https://mlsanity.com/paper/2405.11982/) (2024)
- [Evaluating Robustness of Large Language Models Against Multilingual Typographical Errors](https://mlsanity.com/paper/2510.09536/) (2025)
- [A new Potential-Based Reward Shaping for Reinforcement Learning Agent](https://mlsanity.com/paper/1902.06239/) (2019)

## Built on by (most cited first, 1 of 1)

- [SimVerity: When Does Simulated Agent Success Survive Physical Deployment?](https://mlsanity.com/paper/2608.25067/) (2026)

## Abstract (arXiv)

Foundation model agents are increasingly deployed for real-world decision-making, but suffer from the sim-to-real gap. While robotics and classical control have mature frameworks to address this gap, the foundation model community is treating agent robustness as an entirely novel phenomenon. Our paper proposes formalizing the foundation model agent evaluation and training gap as a classical sim-to-real problem structured entirely around the four elements of a Markov Decision Process, including Observation, Action, Transition, and Reward. In this paper, we set a comprehensive research agenda that translates classical discrepancies into the foundation model domain and advocates for adopting established solutions like domain randomization. We provide concrete examples, such as a multilingual tool calling to demonstrate how severe observation space gaps lead to operationally invalid actions despite correct semantic intent. Ultimately, this agenda aims to drive a paradigm shift, yielding a unified vocabulary and standardized stress test benchmarks to foster a new generation of highly trustworthy agents for reliable real-world applications.