mlsanityData to 7 Oct 2026

Replacing Rewards with Examples: Example-Based Policy Search via Recursive Classification

arXiv 2103.12656 ↗First posted 23 Mar 2021cs.LG cs.ROcited by 38

Not on the map. Why?

Abstract (arXiv)

Reinforcement learning (RL) algorithms assume that users specify tasks by manually writing down a reward function. However, this process can be laborious and demands considerable technical expertise. Can we devise RL algorithms that instead enable users to specify tasks simply by providing examples of successful outcomes? In this paper, we derive a control algorithm that maximizes the future probability of these successful outcome examples. Prior work has approached similar problems with a two-stage process, first learning a reward function and then optimizing this reward function using another RL algorithm. In contrast, our method directly learns a value function from transitions and successful outcomes, without learning this intermediate reward function. Our method therefore requires fewer hyperparameters to tune and lines of code to debug. We show that our method satisfies a new data-driven Bellman equation, where examples take the place of the typical reward function term. Experiments show that our approach outperforms prior methods that learn explicit reward functions.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 28 of its 59 references that are in the map. First posted on arXiv 23 Mar 2021.

Builds on

28 of 59 references.

References · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Continuous control with deep reinforcement learningcited by 3,279
  3. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  4. 2016Generative Adversarial Imitation Learningcited by 1,576
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
  9. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
  10. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  11. 2018Discriminator-Actor-Critic: Addressing Sample Inefficiency and Reward Bias in Adversarial Imitation Learningcited by 203
  12. 2019Imitation Learning via Off-Policy Distribution Matchingcited by 182
16 more
  1. 2015Maximum Entropy Deep Inverse Reinforcement Learningcited by 173
  2. 2015Taming the Noise in Reinforcement Learning via Soft Updatescited by 125
  3. 2019End-to-End Robotic Reinforcement Learning without Reward Engineeringcited by 115
  4. 2018Near-Optimal Representation Learning for Hierarchical Reinforcement Learningcited by 107
  5. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  6. 2016Deep Successor Reinforcement Learningcited by 72
  7. 2020Offline Learning from Demonstrations and Unlabeled Experiencecited by 72
  8. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  9. 2017The Feeling of Success: Does Touch Sensing Help Predict Grasp Outcomes?cited by 65
  10. 2019Task-Relevant Adversarial Imitation Learningcited by 45
  11. 2019Positive-Unlabeled Reward Learningcited by 36
  12. 2018A Practical Approach to Insertion with Variable Socket Position Using Deep Reinforcement Learningcited by 35
  13. 2019Off-Policy Evaluation via Off-Policy Classificationcited by 32
  14. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  15. 2020Semi-supervised reward learning for offline reinforcement learningcited by 22
  16. 2019Reinforcement Learning without Ground-Truth Statecited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 10; 2023: 14; 2024: 6; 2025: 1; 2026: 2.

38 papers.

Sort:

By area:

  1. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  2. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrationscited by 89
  3. 2022Few-Shot Preference Learning for Human-in-the-Loop RLcited by 85
  4. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  5. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  6. 2023Benchmarks and Algorithms for Offline Preference-Based Reward Learningcited by 55
  7. 2023Dual RL: Unification and New Methods for Reinforcement and Imitation Learningcited by 48
  8. 2022Versatile Offline Imitation from Observations and Examples via Regularized State-Occupancy Matchingcited by 43
  9. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  10. 2021Adversarial Intrinsic Motivation for Reinforcement Learningcited by 40
  11. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 24
  12. 2022LobsDICE: Offline Learning from Observation via Stationary Distribution Correction Estimationcited by 22
26 more
  1. 2023MAHALO: Unifying Offline Reinforcement Learning and Imitation Learning from Observationscited by 18
  2. 2021Offline Preference-Based Apprenticeship Learningcited by 17
  3. 2023Outcome-directed Reinforcement Learning by Uncertainty & Temporal Distance-Aware Curriculum Goal Generationcited by 14
  4. 2023f-Policy Gradients: A General Framework for Goal Conditioned RL using f-Divergencescited by 14
  5. 2022TarGF: Learning Target Gradient Field to Rearrange Objects without Explicit Goal Specificationcited by 13
  6. 2021Example-Driven Model-Based Reinforcement Learning for Solving Long-Horizon Visuomotor Taskscited by 12
  7. 2023Diffused Task-Agnostic Milestone Plannercited by 12
  8. 2022Automating Reinforcement Learning with Example-based Resetscited by 11
  9. 2024A Dual Approach to Imitation Learning from Observations with Offline Datasetscited by 10
  10. 2023CLIP-Motion: Learning Reward Functions for Robotic Actions Using Consecutive Observationscited by 8
  11. 2023Learning to Solve Tasks with Exploring Prior Behaviourscited by 4
  12. 2023Diversify & Conquer: Outcome-directed Curriculum RL via Out-of-Distribution Disagreementcited by 4
  13. 2024Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learningcited by 4
  14. 2021Combining Learning from Human Feedback and Knowledge Engineering to Solve Hierarchical Tasks in Minecraftcited by 3
  15. 2023Contrastive Example-Based Controlcited by 3
  16. 2023A Simple Solution for Offline Imitation from Observations and Examples with Possibly Incomplete Trajectoriescited by 3
  17. 2023Demonstration-free Autonomous Reinforcement Learning via Implicit and Bidirectional Curriculumcited by 2
  18. 2024Surgical Task Automation Using Actor-Critic Frameworks and Self-Supervised Imitation Learningcited by 2
  19. 2024Reinforcement Learning From Imperfect Corrective Actions And Proxy Rewardscited by 2
  20. 2026PlatoLTL: Scaling LTL-Guided Multi-Task RLcited by 2
  21. 2022Training Robots to Evaluate Robots: Example-Based Interactive Reward Functions for Policy Learningcited by 1
  22. 2023Synthesizing a Progression of Subtasks for Block-Based Visual Programming Taskscited by 1
  23. 2024Efficient Imitation Without Demonstrations via Value-Penalized Auxiliary Control from Examplescited by 1
  24. 2026Learning from Demonstrations via Capability-Aware Goal Samplingcited by 1
  25. 2024Semi-Supervised One-Shot Imitation Learningcited by 0
  26. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0