Sequential Decision Making with Expert Demonstrations under Unobserved Heterogeneity
Not on the map. Why?
Abstract (arXiv)
We study the problem of online sequential decision-making given auxiliary demonstrations from experts who made their decisions based on unobserved contextual information. These demonstrations can be viewed as solving related but slightly different problems than what the learner faces. This setting arises in many application domains, such as self-driving cars, healthcare, and finance, where expert demonstrations are made using contextual information, which is not recorded in the data available to the learning agent. We model the problem as zero-shot meta-reinforcement learning with an unknown distribution over the unobserved contextual variables and a Bayesian regret minimization objective, where the unobserved variables are encoded as parameters with an unknown prior. We propose the Experts-as-Priors algorithm (ExPerior), an empirical Bayes approach that utilizes expert data to establish an informative prior distribution over the learner's decision-making problem. This prior distribution enables the application of any Bayesian approach for online decision-making, such as posterior sampling. We demonstrate that our strategy surpasses existing behaviour cloning, online, and online-offline baselines for multi-armed bandits, Markov decision processes (MDPs), and partially observable MDPs, showcasing the broad reach and utility of ExPerior in using expert demonstrations across different decision-making setups.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 10 Apr 2024.
Builds on
42 of 63 references.
References · 42
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2015Continuous control with deep reinforcement learningcited by 3,279
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
- 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 723
- 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 685
- 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
- 2016Deep Exploration via Bootstrapped DQNcited by 518
- 2017A Tutorial on Thompson Samplingcited by 420
- 2019Efficient Off-Policy Meta-Reinforcement Learning via Probabilistic Context Variablescited by 372
- 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
- 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 322
- 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 302
30 more
- 2018Learning to Adapt in Dynamic, Real-World Environments Through Meta-Reinforcement Learningcited by 270
- 2013(More) Efficient Reinforcement Learning via Posterior Samplingcited by 233
- 2018Randomized Prior Functions for Deep Reinforcement Learningcited by 218
- 2015Contextual Markov Decision Processescited by 213
- 2014An Information-Theoretic Analysis of Thompson Samplingcited by 178
- 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 158
- 2017Deep Exploration via Randomized Value Functionscited by 155
- 2023Supervised Pretraining Can Learn In-Context Reinforcement Learningcited by 137
- 2019Behaviour Suite for Reinforcement Learningcited by 129
- 2018Meta-Reinforcement Learning of Structured Exploration Strategiescited by 128
- 2023A Tutorial on Meta-Reinforcement Learningcited by 102
- 2019Meta-Inverse Reinforcement Learning with Probabilistic Context Variablescited by 50
- 2020Meta-learning with Stochastic Linear Banditscited by 45
- 2022Leveraging Offline Data in Online Reinforcement Learningcited by 42
- 2020Off-policy Evaluation in Infinite-Horizon Reinforcement Learning with Latent Confounderscited by 40
- 2022Causal Imitation Learning with Unobserved Confounderscited by 40
- 2020Bridging the Imitation Gap by Adaptive Insubordinationcited by 38
- 2017Data-driven Planning via Imitation Learningcited by 36
- 2022Sequence Model Imitation Learning with Unobserved Contextscited by 34
- 2023Approximate Thompson Sampling via Epistemic Neural Networkscited by 31
- 2023Provable and Practical: Efficient Exploration in Reinforcement Learning via Langevin Monte Carlocited by 29
- 2019Guided Meta-Policy Searchcited by 26
- 2019Watch, Try, Learn: Meta-Learning from Demonstrations and Rewardcited by 26
- 2020Robust Asymmetric Learning in POMDPscited by 23
- 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
- 2023TGRL: An Algorithm for Teacher Guided Reinforcement Learningcited by 16
- 2023Leveraging Demonstrations to Improve Online Learning: Quality Matterscited by 13
- 2023Bridging Imitation and Online Reinforcement Learning: An Optimistic Talecited by 5
- 2020Langevin DQNcited by 1
- 2022Zero-shot meta-learning for small-scale data from human subjectscited by 1
Built on by
Citing papers per year: 2025: 1.
1 paper.