# Self-Improving Embodied Foundation Models

arXiv: https://arxiv.org/abs/2509.15155 · HTML: https://mlsanity.com/paper/2509.15155/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Self-evolving LLM agents, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 50 of its 58 references that are in the map.
- First posted on arXiv 18 Sep 2025.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-evolving LLM agents](https://mlsanity.com/synth/self-evolving-llm-agents/)
- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 50)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Flow Matching for Generative Modeling](https://mlsanity.com/paper/2210.02747/) (2022)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Learning to summarize from human feedback](https://mlsanity.com/paper/2009.01325/) (2020)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [π0.5: a Vision-Language-Action Model with Open-World Generalization](https://mlsanity.com/paper/2504.16054/) (2025)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Gemma: Open Models Based on Gemini Research and Technology](https://mlsanity.com/paper/2403.08295/) (2024)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [Image Transformer](https://mlsanity.com/paper/1802.05751/) (2018)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Eureka: Human-Level Reward Design via Coding Large Language Models](https://mlsanity.com/paper/2310.12931/) (2023)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Learning Interactive Real-World Simulators](https://mlsanity.com/paper/2310.06114/) (2023)
- [VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Training](https://mlsanity.com/paper/2210.00030/) (2022)
- [Universal Manipulation Interface: In-The-Wild Robot Teaching Without In-The-Wild Robots](https://mlsanity.com/paper/2402.10329/) (2024)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [Sim-to-Real: Learning Agile Locomotion For Quadruped Robots](https://mlsanity.com/paper/1804.10332/) (2018)
- [Language to Rewards for Robotic Skill Synthesis](https://mlsanity.com/paper/2306.08647/) (2023)
- [PaLI-X: On Scaling up a Multilingual Vision and Language Model](https://mlsanity.com/paper/2305.18565/) (2023)

## Built on by (most cited first, 30 of 32)

- [π*0.6: a VLA That Learns From Experience](https://mlsanity.com/paper/2511.14759/) (2025)
- [Robometer: Scaling General-Purpose Robotic Reward Models via Trajectory Comparisons](https://mlsanity.com/paper/2603.02115/) (2026)
- [Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulation](https://mlsanity.com/paper/2512.23703/) (2025)
- [SIMA 2: A Generalist Embodied Agent for Virtual Worlds](https://mlsanity.com/paper/2512.04797/) (2025)
- [Counterfactual VLA: Self-Reflective Vision-Language-Action Model with Adaptive Reasoning](https://mlsanity.com/paper/2512.24426/) (2025)
- [chi0: Resource-Aware Robust Manipulation via Taming Distributional Inconsistencies](https://mlsanity.com/paper/2602.09021/) (2026)
- [The Last AI Built by Humans: Toward Genuine Recursive Self-Improvement](https://mlsanity.com/paper/2609.11873/) (2026)
- [ViVa: A Video-Generative Value Model for Robot Reinforcement Learning](https://mlsanity.com/paper/2604.08168/) (2026)
- [ALOE: Action-Level Off-Policy Evaluation for Vision-Language-Action Model Post-Training](https://mlsanity.com/paper/2602.12691/) (2026)
- [Safety in Embodied AI: A Survey of Risks, Attacks, and Defenses](https://mlsanity.com/paper/2605.02900/) (2026)
- [Self-Improvement in Multimodal Large Language Models: A Survey](https://mlsanity.com/paper/2510.02665/) (2025)
- [IG-RFT: An Interaction-Guided RL Framework for VLA Models in Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2602.20715/) (2026)
- [TapSampling: Inference-Time Sampling with a Task-Progress-Understanding Verifier for Robotic Manipulation](https://mlsanity.com/paper/2605.25547/) (2026)
- [ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI](https://mlsanity.com/paper/2607.04426/) (2026)
- [Success Conditioning as Policy Improvement: The Optimization Problem Solved by Imitating Success](https://mlsanity.com/paper/2601.18175/) (2026)
- [Self-Correcting VLA: Online Action Refinement via Sparse World Imagination](https://mlsanity.com/paper/2602.21633/) (2026)
- [RARM: Confidence-Gated Progress Reward Modeling for RL in Manipulation](https://mlsanity.com/paper/2606.22027/) (2026)
- [WorldSample: Closed-loop Real-robot RL with World Modelling](https://mlsanity.com/paper/2607.02431/) (2026)
- [Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worlds](https://mlsanity.com/paper/2603.18532/) (2026)
- [ExToken: Structured Exploration for Efficient Vision-Language-Action Reinforcement Fine-tuning](https://mlsanity.com/paper/2607.12931/) (2026)
- [In-Context Learning for Robots: Methods and Applications](https://mlsanity.com/paper/2609.36012/) (2026)
- [Feat2Go: Visual Feature-Grounded Value Estimation for Embodied Reinforcement Learning](https://mlsanity.com/paper/2605.30795/) (2026)
- [Completion at the Boundary (CaB): Deployable Switching with Completion-Aware Control under Limited Calibration](https://mlsanity.com/paper/2606.00145/) (2026)
- [Temporal Self-Imitation Learning](https://mlsanity.com/paper/2606.19752/) (2026)
- [Vision Language Action (VLA) Models for Unmanned Aerial Robotics and Bimanual Manipulation: A Review](https://mlsanity.com/paper/2607.06706/) (2026)
- [HELP: Human-Efficient Large-Scale Robot Post-Training with Rollout Segmentation](https://mlsanity.com/paper/2607.09776/) (2026)
- [TrustRoboReward: Preference-Ordered Isotonic Score Editing for Multi-Paradigm Robot Reward Models](https://mlsanity.com/paper/2608.08491/) (2026)
- [Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation](https://mlsanity.com/paper/2608.19490/) (2026)
- [No Free Checker: A Survey of Verifiers for Robot Policies](https://mlsanity.com/paper/2609.09250/) (2026)
- [Imagine-RL: Residual-Confidence-Guided Cross-Attention for World-Model-Augmented VLA Reinforcement Learning](https://mlsanity.com/paper/2609.24033/) (2026)

## Abstract (arXiv)

Foundation models trained on web-scale data have revolutionized robotics, but their application to low-level control remains largely limited to behavioral cloning. Drawing inspiration from the success of the reinforcement learning stage in fine-tuning large language models, we propose a two-stage post-training approach for robotics. The first stage, Supervised Fine-Tuning (SFT), fine-tunes pretrained foundation models using both: a) behavioral cloning, and b) steps-to-go prediction objectives. In the second stage, Self-Improvement, steps-to-go prediction enables the extraction of a well-shaped reward function and a robust success detector, enabling a fleet of robots to autonomously practice downstream tasks with minimal human supervision. Through extensive experiments on real-world and simulated robot embodiments, our novel post-training recipe unveils significant results on Embodied Foundation Models. First, we demonstrate that the combination of SFT and Self-Improvement is significantly more sample-efficient than scaling imitation data collection for supervised learning, and that it leads to policies with significantly higher success rates. Further ablations highlight that the combination of web-scale pretraining and Self-Improvement is the key to this sample-efficiency. Next, we demonstrate that our proposed combination uniquely unlocks a capability that current methods cannot achieve: autonomously practicing and acquiring novel skills that generalize far beyond the behaviors observed in the imitation learning datasets used during training. These findings highlight the transformative potential of combining pretrained foundation models with online Self-Improvement to enable autonomous skill acquisition in robotics. Our project website can be found at https://self-improving-efms.github.io .