mlsanityData to 7 Oct 2026

Behavior Estimation from Multi-Source Data for Offline Reinforcement Learning

arXiv 2211.16078 ↗First posted 29 Nov 2022cs.LG cs.ROcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) have received rising interest due to its appealing data efficiency. The present study addresses behavior estimation, a task that lays the foundation of many offline RL algorithms. Behavior estimation aims at estimating the policy with which training data are generated. In particular, this work considers a scenario where the data are collected from multiple sources. In this case, neglecting data heterogeneity, existing approaches for behavior estimation suffers from behavior misspecification. To overcome this drawback, the present study proposes a latent variable model to infer a set of policies from data, which allows an agent to use as behavior policy the policy that best describes a particular trajectory. This model provides with a agent fine-grained characterization for multi-source data and helps it overcome behavior misspecification. This work also proposes a learning algorithm for this model and illustrates its practical usage via extending an existing offline RL algorithm. Lastly, with extensive evaluation this work confirms the existence of behavior misspecification and the efficacy of the proposed model.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 23 of its 33 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

23 of 33 references.

Closest ancestors · 7

Same direction, newest first

  1. 2021Model-Based Offline Planning with Trajectory Pruning16 May
  2. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  3. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  7. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 16

  1. 2016Layer Normalizationcited by 5,579
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  5. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  6. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  7. 2019DualDICE: Behavior-Agnostic Estimation of Discounted Stationary Distribution Correctionscited by 254
  8. 2019Batch Policy Learning under Constraintscited by 242
  9. 2018RoboTurk: A Crowdsourcing Platform for Robotic Skill Learning through Imitationcited by 202
  10. 2018Top-K Off-Policy Correction for a REINFORCE Recommender Systemcited by 181
  11. 2019Imitation Learning from Imperfect Demonstrationcited by 112
  12. 2018SOM-VAE: Interpretable Discrete Representation Learning on Time Seriescited by 59
4 more
  1. 2017Consistent On-Line Off-Policy Evaluationcited by 51
  2. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  3. 2019Off-Policy Policy Gradient with State Distribution Correctioncited by 43
  4. 2020Reducing Sampling Error in Batch Temporal Difference Learningcited by 11

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 0; 2025: 1.

2 papers.

  1. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularizationcited by 3
  2. 2025Behaviour Discovery and Attribution for Explainable Reinforcement Learningcited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2022Offline Reinforcement Learning with Adaptive Behavior Regularization
  2. 2023Iteratively Refined Behavior Regularization for Offline Reinforcement Learning
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RL
  5. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  6. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning
  7. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  8. 2023Matrix Estimation for Offline Reinforcement Learning with Low-Rank Structure