mlsanityData to 7 Oct 2026

On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfer

arXiv 2312.03673 ↗First posted 6 Dec 2023cs.RO cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

We study the choice of action space in robot manipulation learning and sim-to-real transfer. We define metrics that assess the performance, and examine the emerging properties in the different action spaces. We train over 250 reinforcement learning~(RL) agents in simulated reaching and pushing tasks, using 13 different control spaces. The choice of spaces spans combinations of common action space design characteristics. We evaluate the training performance in simulation and the transfer to a real-world environment. We identify good and bad characteristics of robotic action spaces and make recommendations for future designs. Our findings have important implications for the design of RL algorithms for robot manipulation tasks, and highlight the need for careful consideration of action spaces when training and transferring RL agents for real-world robotics.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 38 of its 45 references that are in the map. First posted on arXiv 6 Dec 2023.

Builds on

38 of 45 references.

References · 38

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  4. 2018On the Continuity of Rotation Representations in Neural Networkscited by 1,186
  5. 2021Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learningcited by 882
  6. 2019Solving Rubik's Cube with a Robot Handcited by 699
  7. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  8. 2019Learning agile and dynamic motor skills for legged robotscited by 402
  9. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  10. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  11. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 277
  12. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 203
26 more
  1. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  2. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  3. 2019Making Sense of Vision and Touch: Learning Multimodal Representations for Contact-Rich Taskscited by 80
  4. 2020NeuralSim: Augmenting Differentiable Simulators with Neural Networkscited by 77
  5. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  6. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  7. 2019Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Taskscited by 56
  8. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 56
  9. 2021Robot Learning from Randomized Simulations: A Reviewcited by 56
  10. 2015From Pixels to Torques: Policy Learning with Deep Dynamical Modelscited by 44
  11. 2016Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?cited by 44
  12. 2019Efficient Bimanual Manipulation Using Learned Task Schemascited by 44
  13. 2023IndustReal: Transferring Contact-Rich Assembly Tasks from Simulation to Realitycited by 44
  14. 2019Reinforcement Learning on Variable Impedance Controller for High-Precision Robotic Assemblycited by 33
  15. 2021LASER: Learning a Latent Action Space for Efficient Reinforcement Learningcited by 32
  16. 2022A Benchmark Comparison of Learned Control Policies for Agile Quadrotor Flightcited by 32
  17. 2020Learning to Compose Hierarchical Object-Centric Controllers for Robotic Manipulationcited by 17
  18. 2019A Comparison of Action Spaces for Learning Manipulation Taskscited by 16
  19. 2019Learning Variable Impedance Control for Contact Sensitive Taskscited by 15
  20. 2020Learning Task Space Actions for Bipedal Locomotioncited by 15
  21. 2022Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learningcited by 15
  22. 2020Learning Vision-based Reactive Policies for Obstacle Avoidancecited by 10
  23. 2020Augmenting Differentiable Simulators with Neural Networks to Close the Sim2Real Gapcited by 9
  24. 2021Learning to Centralize Dual-Arm Assemblycited by 9
  25. 2022CLAS: Coordinating Multi-Robot Manipulation with Central Latent Action Spacescited by 6
  26. 2021Learning Robotic Manipulation Skills Using an Adaptive Force-Impedance Action Spacecited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 8; 2026: 5.

20 papers.

Sort:

By area:

  1. 2024TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correctioncited by 50
  2. 2024A Survey of Embodied Learning for Object-Centric Robotic Manipulationcited by 34
  3. 2025The Reality Gap in Robotics: Challenges, Solutions, and Best Practicescited by 22
  4. 2025Accelerating Model-Based Reinforcement Learning with State-Space World Modelscited by 10
  5. 2024Automatic Environment Shaping is the Next Frontier in RLcited by 6
  6. 2024Student-Informed Teacher Trainingcited by 6
  7. 2024LIMT: Language-Informed Multi-Task Visual World Modelscited by 4
  8. 2025SLIM: Sim-to-Real Legged Instructive Manipulation via Long-Horizon Visuomotor Learningcited by 4
  9. 2025A Monocular Event-Camera Motion Capture Systemcited by 3
  10. 2025How Well do Diffusion Policies Learn Kinematic Constraint Manifolds?cited by 3
  11. 2024The Shortcomings of Force-from-Motion in Robot Learningcited by 1
  12. 2026Approximate Imitation Learning for Event-based Quadrotor Flight in Cluttered Environmentscited by 1
8 more
  1. 2024Integrating DeepRL with Robust Low-Level Control in Robotic Manipulators for Non-Repetitive Reaching Taskscited by 0
  2. 2025Rapidly Learning Soft Robot Control via Implicit Time-Steppingcited by 0
  3. 2025Push Smarter, Not Harder: Hierarchical RL-Diffusion Policy for Efficient Nonprehensile Manipulationcited by 0
  4. 2025Kinematics-Aware Diffusion Policy with Consistent 3D Observation and Action Space for Whole-Arm Robotic Manipulationcited by 0
  5. 2026Behavior Cloning Under PD Control: A Finite-Horizon Theory of Gain-Dependent Error Amplificationcited by 0
  6. 2026APEX: Adaptive Policy Execution for Precise Manipulationcited by 0
  7. 2026Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulationcited by 0
  8. 2026ASGARD: Action-Space Guard for UAV Resilience via Reinforcement Learningcited by 0