mlsanityData to 14 Oct 2026

Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulation

arXiv 2606.18594 ↗First posted 17 Jun 2026cs.RO cs.AIcited by 1

On the map: Deep reinforcement learning methods → RL for robot locomotion and manipulation

Abstract (arXiv)

In real-world reinforcement learning (RL), the choice of action space can play a key role in shaping motion smoothness, safety, and overall task performance. In this study, we evaluate pose increment, pose velocity, joint position increment, and joint velocity across two vision-based manipulation tasks: object picking and pushing. We train policies in simulation and deploy them to the real world using sim-to-real transfer. We find that action-space representation indeed significantly affects sim-to-real performance. In particular, we find that the joint velocity action space is best for the vision-based picking and pushing tasks in terms of smoothness and final task performance. We also provide practical guidance for RL practitioners in choosing action spaces for both simulation and real-world experiments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RL for robot locomotion and manipulation, one of 15 directions in Deep reinforcement learning methods. Builds on 25 of its 28 references that are in the map. First posted on arXiv 17 Jun 2026.

Builds on

25 of 28 references.

References · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  3. 2019Solving Rubik's Cube with a Robot Handcited by 734
  4. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 672
  5. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  6. 2020dm_control: Software and Tasks for Continuous Controlcited by 375
  7. 2018Sim-to-Real: Learning Agile Locomotion For Quadruped Robotscited by 319
  8. 2016Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collectioncited by 316
  9. 2018Closing the Sim-to-Real Loop: Adapting Simulation Randomization with Real World Experiencecited by 217
  10. 2018Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Taskscited by 154
  11. 2021Understanding Domain Randomization for Sim-to-real Transfercited by 80
  12. 2025MuJoCo Playgroundcited by 78
13 more
  1. 2019Variable Impedance Control in End-Effector Space: An Action Space for Reinforcement Learning in Contact-Rich Taskscited by 67
  2. 2018Benchmarking Reinforcement Learning Algorithms on Real-World Robotscited by 65
  3. 2016Learning Locomotion Skills Using DeepRL: Does the Choice of Action Space Matter?cited by 48
  4. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 33
  5. 2022Learning Torque Control for Quadrupedal Locomotioncited by 29
  6. 2023DexPBT: Scaling up Dexterous Manipulation for Hand-Arm Systems with Population Based Trainingcited by 29
  7. 2023On the Role of the Action Space in Robot Manipulation Learning and Sim-to-Real Transfercited by 26
  8. 2023Torque-based Deep Reinforcement Learning for Task-and-Robot Agnostic Learning on Bipedal Robots Using Sim-to-Real Transfercited by 22
  9. 2019A Comparison of Action Spaces for Learning Manipulation Taskscited by 17
  10. 2023Investigating the Impact of Action Representations in Policy Gradient Algorithmscited by 2
  11. 2023Open-Source Reinforcement Learning Environments Implemented in MuJoCo with Franka Manipulatorcited by 2
  12. 2026Performance Variation in Deep Reinforcement Learningcited by 2
  13. 2025Dynamic Object Masks as Goal Representations for Visual Goal-Conditioned Reinforcement Learningcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Revisiting Temporal Regularization for Smooth Control in Deep Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Grounding Sim-to-Real Generalization in Robotic Manipulation: An Empirical Study with Vision-Language-Action Models
  2. 2022How to Spend Your Robot Time: Bridging Kickstarting and Offline Reinforcement Learning for Vision-based Robotic Manipulation
  3. 2022Implicit Kinematic Policies: Unifying Joint and Cartesian Action Spaces in End-to-End Robot Learning
  4. 2026World Models for Robotic Manipulation: A Survey
  5. 2025ManipLVM-R1: Reinforcement Learning for Reasoning in Embodied Manipulation with Large Vision-Language Models
  6. 2025A comparison of visual representations for real-world reinforcement learning in the context of vacuum gripping
  7. 2024Precise and Dexterous Robotic Manipulation via Human-in-the-Loop Reinforcement Learning
  8. 2026Beyond Pixels: Learning Invariant Rewards for Real-World Robotics From a Few Demonstrations