# Real-to-Sim: Predicting Residual Errors of Robotic Systems with Sparse Data using a Learning-based Unscented Kalman Filter

arXiv: https://arxiv.org/abs/2209.03210 · HTML: https://mlsanity.com/paper/2209.03210/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 23 references that are in the map.
- First posted on arXiv 7 Sep 2022.

## Builds on (most cited first, 11 of 11)

- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey](https://mlsanity.com/paper/2009.13303/) (2020)
- [Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experience](https://mlsanity.com/paper/1810.05687/) (2018)
- [Active Domain Randomization](https://mlsanity.com/paper/1904.04762/) (2019)
- [Auto-Tuned Sim-to-Real Transfer](https://mlsanity.com/paper/2104.07662/) (2021)
- [Self-Supervised Sim-to-Real Adaptation for Visual Robotic Manipulation](https://mlsanity.com/paper/1910.09470/) (2019)
- [SABER: Data-Driven Motion Planner for Autonomously Navigating Heterogeneous Robots](https://mlsanity.com/paper/2108.01262/) (2021)
- [Automated Controller Calibration by Kalman Filtering](https://mlsanity.com/paper/2111.10832/) (2021)
- [SCALER: A Tough Versatile Quadruped Free-Climber Robot](https://mlsanity.com/paper/2207.01180/) (2022)
- [Risk-Averse MPC via Visual-Inertial Input and Recurrent Networks for Online Collision Avoidance](https://mlsanity.com/paper/2007.14035/) (2020)
- [Adaptive Force Controller for Contact-Rich Robotic Systems using an Unscented Kalman Filter](https://mlsanity.com/paper/2207.01033/) (2022)

## Built on by (most cited first, 5 of 5)

- [Adapting World Models with Latent-State Dynamics Residuals](https://mlsanity.com/paper/2504.02252/) (2025)
- [Bridging the Sim-to-Real Gap with Bayesian Inference](https://mlsanity.com/paper/2403.16644/) (2024)
- [OptiState: State Estimation of Legged Robots using Gated Networks with Transformer-based Vision and Kalman Filtering](https://mlsanity.com/paper/2401.16719/) (2024)
- [Simulation Priors for Data-Efficient Deep Learning](https://mlsanity.com/paper/2509.05732/) (2025)
- [Learning What Matters: Adaptive Information-Theoretic Objectives for Robot Exploration](https://mlsanity.com/paper/2605.12084/) (2026)

## Abstract (arXiv)

Achieving highly accurate dynamic or simulator models that are close to the real robot can facilitate model-based controls (e.g., model predictive control or linear-quadradic regulators), model-based trajectory planning (e.g., trajectory optimization), and decrease the amount of learning time necessary for reinforcement learning methods. Thus, the objective of this work is to learn the residual errors between a dynamic and/or simulator model and the real robot. This is achieved using a neural network, where the parameters of a neural network are updated through an Unscented Kalman Filter (UKF) formulation. Using this method, we model these residual errors with only small amounts of data -- a necessity as we improve the simulator/dynamic model by learning directly from real-world operation. We demonstrate our method on robotic hardware (e.g., manipulator arm, and a wheeled robot), and show that with the learned residual errors, we can further close the reality gap between dynamic models, simulations, and actual hardware.