mlsanityData to 7 Oct 2026

Autonomous Robotic Reinforcement Learning with Asynchronous Human Feedback

arXiv 2310.20608 ↗First posted 31 Oct 2023cs.LG cs.AI cs.ROcited by 12

Not on the map. Why?

Abstract (arXiv)

Ideally, we would place a robot in a real-world environment and leave it there improving on its own by gathering more experience autonomously. However, algorithms for autonomous robotic learning have been challenging to realize in the real world. While this has often been attributed to the challenge of sample complexity, even sample-efficient techniques are hampered by two major challenges - the difficulty of providing well "shaped" rewards, and the difficulty of continual reset-free training. In this work, we describe a system for real-world reinforcement learning that enables agents to show continual improvement by training directly in the real world without requiring painstaking effort to hand-design reward functions or reset mechanisms. Our system leverages occasional non-expert human-in-the-loop feedback from remote users to learn informative distance functions to guide exploration while leveraging a simple self-supervised learning algorithm for goal-directed policy learning. We show that in the absence of resets, it is particularly important to account for the current "reachability" of the exploration policy when deciding which regions of the space to explore. Based on this insight, we instantiate a practical learning system - GEAR, which enables robots to simply be placed in real-world environments and left to train autonomously without interruption. The system streams robot experience to a web interface only requiring occasional asynchronous feedback from remote, crowdsourced, non-expert humans in the form of binary comparative feedback. We evaluate this system on a suite of robotic tasks in simulation and demonstrate its effectiveness at learning behaviors both in simulation and the real world. Project website https://guided-exploration-autonomous-rl.github.io/GEAR/.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 54 of its 72 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

54 of 72 references.

References · 54

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2016Density estimation using Real NVPcited by 2,029
  6. 2017Hindsight Experience Replaycited by 1,096
  7. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  8. 2016Pixel Recurrent Neural Networkscited by 814
  9. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  10. 2022DayDreamer: World Models for Physical Robot Learningcited by 372
  11. 2019Learning Latent Plans from Playcited by 310
  12. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 298
38 more
  1. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 298
  2. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  3. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
  4. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  5. 2022Interactive Language: Talking to Robots in Real Timecited by 217
  6. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  7. 2022Goal-Conditioned Reinforcement Learning: Problems and Solutionscited by 145
  8. 2022DeXtreme: Transfer of Agile In-hand Manipulation from Simulation to Realitycited by 136
  9. 2019TossingBot: Learning to Throw Arbitrary Objects with Residual Physicscited by 131
  10. 2020The Ingredients of Real-World Robotic Reinforcement Learningcited by 111
  11. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  12. 2021Reward (Mis)design for Autonomous Drivingcited by 102
  13. 2016Neural Autoregressive Distribution Estimationcited by 94
  14. 2021Reset-Free Reinforcement Learning via Multi-Task Learning: Learning Dexterous Manipulation Behaviors without Human Interventioncited by 90
  15. 2018Batch Active Preference-Based Learning of Reward Functionscited by 87
  16. 2017Leave no Trace: Learning to Reset for Safe and Autonomous Reinforcement Learningcited by 86
  17. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 79
  18. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  19. 2023Inverse Preference Learning: Preference-based RL without a Reward Functioncited by 68
  20. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  21. 2021Autonomous Reinforcement Learning: Formalism and Benchmarkingcited by 33
  22. 2021Autonomous Reinforcement Learning via Subgoal Curriculacited by 32
  23. 2020Continual Learning of Control Primitives: Skill Discovery via Reset-Gamescited by 29
  24. 2023Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learningcited by 29
  25. 2019Deep Bayesian Reward Learning from Preferencescited by 28
  26. 2019Learning to Reach Goals via Iterated Supervised Learningcited by 26
  27. 2020Feature Expansive Reward Learning: Rethinking Human Inputcited by 25
  28. 2022A State-Distribution Matching Approach to Non-Episodic Reinforcement Learningcited by 21
  29. 2022Dexterous Manipulation from Images: Autonomous Real-World RL via Substep Guidancecited by 20
  30. 2021Physical Interaction as Communication: Learning Robot Objectives Online from Human Correctionscited by 19
  31. 2023Diagnosis, Feedback, Adaptation: A Human-in-the-Loop Framework for Test-Time Policy Adaptationcited by 16
  32. 2021Situational Confidence Assistance for Lifelong Shared Autonomycited by 15
  33. 2022Demonstration-Bootstrapped Autonomous Practicing via Multi-Task Reinforcement Learningcited by 15
  34. 2018Learning under Misspecified Objective Spacescited by 14
  35. 2021Wish you were here: Hindsight Goal Selection for long-horizon dexterous manipulationcited by 13
  36. 2023Breadcrumbs to the Goal: Goal-Conditioned Exploration from Human-in-the-Loop Feedbackcited by 13
  37. 2019Understanding Teacher Gaze Patterns for Robot Learningcited by 12
  38. 2018A Data-Efficient Framework for Training and Sim-to-Real Transfer of Navigation Policiescited by 10

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 7; 2025: 2; 2026: 3.

12 papers.

  1. 2024Reconciling Reality through Simulation: A Real-to-Sim-to-Real Approach for Robust Manipulationcited by 135
  2. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 20
  3. 2024Robot Learning with Super-Linear Scalingcited by 10
  4. 2024How Useful is Intermittent, Asynchronous Expert Feedback for Bayesian Optimization?cited by 2
  5. 2024Coaching a Robotic Sonographer: Learning Robotic Ultrasound with Sparse Expert's Feedbackcited by 2
  6. 2024A General Control-Theoretic Approach for Reinforcement Learning: Theory and Algorithmscited by 1
  7. 2025Residual Reward Models: Leveraging Prior Knowledge for Efficient Preference-based Reinforcement Learning in Roboticscited by 1
  8. 2026Freeform Preference Learning for Robotic Manipulationcited by 1
  9. 2024Growing from Exploration: A self-exploring framework for robots based on foundation modelscited by 0
  10. 2025HAPI: A Model for Learning Robot Facial Expressions from Human Preferencescited by 0
  11. 2026Generating Natural and Expressive Robot Gestures through Iterative Reinforcement Learning with Human Feedback using LLMscited by 0
  12. 2026From Pretraining to Proficiency: Real-World Subtask RL for Long-Horizon Manipulation with Minimal Human Interventioncited by 0