# Modelling Generalized Forces with Reinforcement Learning for Sim-to-Real Transfer

arXiv: https://arxiv.org/abs/1910.09471 · HTML: https://mlsanity.com/paper/1910.09471/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Oct 2019.

## Built on by (most cited first, 5 of 5)

- [Nonprehensile Planar Manipulation through Reinforcement Learning with Multimodal Categorical Exploration](https://mlsanity.com/paper/2308.02459/) (2023)
- [Learning Visuotactile Estimation and Control for Non-prehensile Manipulation under Occlusions](https://mlsanity.com/paper/2412.13157/) (2024)
- [A Simple Approach to Continual Learning by Transferring Skill Parameters](https://mlsanity.com/paper/2110.10255/) (2021)
- [Physics-Informed Model and Hybrid Planning for Efficient Dyna-Style Reinforcement Learning](https://mlsanity.com/paper/2407.02217/) (2024)
- [Sim-to-Real of Humanoid Locomotion Policies via Joint Torque Space Perturbation Injection](https://mlsanity.com/paper/2603.21853/) (2026)

## Abstract (arXiv)

Learning robotic control policies in the real world gives rise to challenges in data efficiency, safety, and controlling the initial condition of the system. On the other hand, simulations are a useful alternative as they provide an abundant source of data without the restrictions of the real world. Unfortunately, simulations often fail to accurately model complex real-world phenomena. Traditional system identification techniques are limited in expressiveness by the analytical model parameters, and usually are not sufficient to capture such phenomena. In this paper we propose a general framework for improving the analytical model by optimizing state dependent generalized forces. State dependent generalized forces are expressive enough to model constraints in the equations of motion, while maintaining a clear physical meaning and intuition. We use reinforcement learning to efficiently optimize the mapping from states to generalized forces over a discounted infinite horizon. We show that using only minutes of real world data improves the sim-to-real control policy transfer. We demonstrate the feasibility of our approach by validating it on a nonprehensile manipulation task on the Sawyer robot.