mlsanityData to 14 Oct 2026

Efficient Meta Reinforcement Learning for Preference-based Fast Adaptation

arXiv 2211.10861 ↗First posted 20 Nov 2022cs.LG cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Learning new task-specific skills from a few trials is a fundamental challenge for artificial intelligence. Meta reinforcement learning (meta-RL) tackles this problem by learning transferable policies that support few-shot adaptation to unseen tasks. Despite recent advances in meta-RL, most existing methods require the access to the environmental reward function of new tasks to infer the task objective, which is not realistic in many practical applications. To bridge this gap, we study the problem of few-shot adaptation in the context of human-in-the-loop reinforcement learning. We develop a meta-RL algorithm that enables fast policy adaptation with preference-based feedback. The agent can adapt to new tasks by querying human's preference between behavior trajectories instead of using per-step numeric rewards. By extending techniques from information theory, our approach can design query sequences to maximize the information gain from human interactions while tolerating the inherent error of non-expert human oracle. In experiments, we extensively evaluate our method, Adaptation with Noisy OracLE (ANOLE), on a variety of meta-RL benchmark tasks and demonstrate substantial improvement over baseline algorithms in terms of both feedback efficiency and error tolerance.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 37 of its 68 references that are in the map. First posted on arXiv 20 Nov 2022.

Builds on

37 of 68 references.

References · 37

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  6. 2016Matching Networks for One Shot Learningcited by 2,255
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  9. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  10. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  11. 2016RL2: Fast Reinforcement Learning via Slow Reinforcement Learningcited by 503
  12. 2018Meta-Learning with Latent Embedding Optimizationcited by 476
25 more
  1. 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 389
  2. 2018Reward learning from human preferences and demonstrations in Ataricited by 316
  3. 2019Learning Latent Plans from Playcited by 316
  4. 2021PEBBLE: Feedback-Efficient Interactive Reinforcement Learning via Relabeling Experience and Unsupervised Pre-trainingcited by 302
  5. 2013Learning to Optimize Via Posterior Samplingcited by 256
  6. 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
  7. 2019VariBAD: A Very Good Method for Bayes-Adaptive Deep RL via Meta-Learningcited by 217
  8. 2019Skew-Fit: State-Covering Self-Supervised Reinforcement Learningcited by 174
  9. 2018Meta-Learning Probabilistic Inference For Predictioncited by 110
  10. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  11. 2019Infinite Mixture Prototypes for Few-Shot Learningcited by 99
  12. 2022SURF: Semi-supervised Reward Learning with Data Augmentation for Feedback-efficient Preference-based Reinforcement Learningcited by 97
  13. 2018ProMP: Proximal Meta-Policy Searchcited by 91
  14. 2021On the Expressivity of Markov Rewardcited by 71
  15. 2019Meta-Q-Learningcited by 62
  16. 2022Reward Uncertainty for Exploration in Preference-based Reinforcement Learningcited by 61
  17. 2020Decoupling Exploration and Exploitation for Meta-Reinforcement Learning without Sacrificescited by 59
  18. 2020Widening the Pipeline in Human-Guided Reinforcement Learning with Explanation and Context-Aware Data Augmentationcited by 42
  19. 2018Unsupervised Meta-Learning for Reinforcement Learningcited by 39
  20. 2021Learning Long-Term Reward Redistribution via Randomized Return Decompositioncited by 33
  21. 2020MetaCURE: Meta Reinforcement Learning with Empowerment-Driven Explorationcited by 25
  22. 2020On the Global Optimality of Model-Agnostic Meta-Learningcited by 21
  23. 2020Personalized Adaptive Meta Learning for Cold-start User Preference Predictioncited by 15
  24. 2021Provable Hierarchy-Based Meta-Reinforcement Learningcited by 5
  25. 2020HMRL: Hyper-Meta Learning for Sparse Reward Reinforcement Learning Problemcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 1; 2025: 1.

4 papers.

  1. 2023Offline Meta Reinforcement Learning with In-Distribution Online Adaptationcited by 12
  2. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  3. 2025Task-Aware Virtual Training: Enhancing Generalization in Meta-Reinforcement Learning for Out-of-Distribution Taskscited by 3
  4. 2023On First-Order Meta-Reinforcement Learning with Moreau Envelopescited by 2