mlsanityData to 7 Oct 2026

State Representations as Incentives for Reinforcement Learning Agents: A Sim2Real Analysis on Robotic Grasping

arXiv 2309.11984 ↗First posted 21 Sep 2023cs.RO cs.AIcited by 1

Not on the map. Why?

Abstract (arXiv)

Choosing an appropriate representation of the environment for the underlying decision-making process of the reinforcement learning agent is not always straightforward. The state representation should be inclusive enough to allow the agent to informatively decide on its actions and disentangled enough to simplify policy training and the corresponding sim2real transfer. Given this outlook, this work examines the effect of various representations in incentivizing the agent to solve a specific robotic task: antipodal and planar object grasping. A continuum of state representations is defined, starting from hand-crafted numerical states to encoded image-based representations, with decreasing levels of induced task-specific knowledge. The effects of each representation on the ability of the agent to solve the task in simulation and the transferability of the learned policy to the real robot are examined and compared against a model-based approach with complete system knowledge. The results show that reinforcement learning agents using numerical states can perform on par with non-learning baselines. Furthermore, we find that agents using image-based representations from pre-trained environment embedding vectors perform better than end-to-end trained agents, and hypothesize that separation of representation learning from reinforcement learning can benefit sim2real transfer. Finally, we conclude that incentivizing the state representation with task-specific knowledge facilitates faster convergence for agent training and increases success rates in sim2real robot control.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 27 of its 44 references that are in the map. First posted on arXiv 21 Sep 2023.

Builds on

27 of 44 references.

References · 27

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Optuna: A Next-generation Hyperparameter Optimization Frameworkcited by 2,455
  4. 2016OpenAI Gymcited by 1,684
  5. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  6. 2018DeepMind Control Suitecited by 935
  7. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  8. 2020Reinforcement Learning with Augmented Datacited by 452
  9. 2020Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Surveycited by 388
  10. 2020FLAMBE: Structural Complexity and Representation Learning of Low Rank MDPscited by 232
  11. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  12. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
15 more
  1. 2018State Representation Learning for Control: An Overviewcited by 125
  2. 2019Self-Supervised Correspondence in Visuomotor Policy Learningcited by 106
  3. 2021Accelerating Robotic Reinforcement Learning via Parameterized Action Primitivescited by 63
  4. 2021Agent Incentives: A Causal Perspectivecited by 51
  5. 2022Efficient Reinforcement Learning in Block MDPs: A Model-free Representation Learning Approachcited by 51
  6. 2019Contextual Imagined Goals for Self-Supervised Robotic Learningcited by 40
  7. 2022Guarantees for Epsilon-Greedy Reinforcement Learning with Function Approximationcited by 29
  8. 2016Towards MRI-Based Autonomous Robotic US Acquisitions: A First Feasibility Studycited by 28
  9. 2020Robotic Grasping using Deep Reinforcement Learningcited by 27
  10. 2020Learning Visual Robotic Control Efficiently with Contrastive Pre-training and Data Augmentationcited by 20
  11. 2021Jerk-limited Real-time Trajectory Generation with Arbitrary Target Statescited by 12
  12. 2022Analysis of Randomization Effects on Sim2Real Transfer in Reinforcement Learning for Robotic Manipulation Taskscited by 12
  13. 2020DeepURL: Deep Pose Estimation Framework for Underwater Relative Localizationcited by 9
  14. 2017Unsupervised state representation learning with robotic priors: a robustness benchmarkcited by 5
  15. 2022Scalable Representation Learning in Linear Contextual Bandits with Constant Regret Guaranteescited by 5

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Safe Continual Domain Adaptation after Sim2Real Transfer of Reinforcement Learning Policies in Roboticscited by 1