mlsanityData to 7 Oct 2026

Learning on One Mode: Addressing Multi-modality in Offline Reinforcement Learning

arXiv 2412.03258 ↗First posted 4 Dec 2024cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

Offline reinforcement learning (RL) seeks to learn optimal policies from static datasets without interacting with the environment. A common challenge is handling multi-modal action distributions, where multiple behaviours are represented in the data. Existing methods often assume unimodal behaviour policies, leading to suboptimal performance when this assumption is violated. We propose weighted imitation Learning on One Mode (LOM), a novel approach that focuses on learning from a single, promising mode of the behaviour policy. By using a Gaussian mixture model to identify modes and selecting the best mode based on expected returns, LOM avoids the pitfalls of averaging over conflicting actions. Theoretically, we show that LOM improves performance while maintaining simplicity in policy learning. Empirically, LOM outperforms existing methods on standard D4RL benchmarks and demonstrates its effectiveness in complex, multi-modal scenarios.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 38 references that are in the map. First posted on arXiv 4 Dec 2024.

Builds on

28 of 38 references.

References · 28

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2015Trust Region Policy Optimizationcited by 2,503
  3. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  4. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  5. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  7. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  11. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  12. 2019Learning Latent Plans from Playcited by 310
16 more
  1. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemblecited by 297
  2. 2020Critic Regularized Regressioncited by 269
  3. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  4. 2021Offline RL Without Off-Policy Evaluationcited by 150
  5. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  6. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learningcited by 125
  7. 2022Rethinking Goal-conditioned Supervised Learning and Its Connection to Offline RLcited by 89
  8. 2022How Far I'll Go: Offline Goal-Conditioned Reinforcement Learning via f-Advantage Regressioncited by 65
  9. 2023What is Essential for Unseen Goal Generalization of Offline Goal-conditioned RL?cited by 36
  10. 2022Let Offline RL Flow: Training Conservative Agents in the Latent Space of Normalizing Flowscited by 21
  11. 2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
  12. 2023Supported Trust Region Optimization for Offline Reinforcement Learningcited by 20
  13. 2023Balancing policy constraint and ensemble size in uncertainty-based offline reinforcement learningcited by 12
  14. 2022A Behavior Regularized Implicit Policy for Offline Reinforcement Learningcited by 11
  15. 2022TD3 with Reverse KL Regularizer for Offline Reinforcement Learning from Mixed Datasetscited by 4
  16. 2023Goal-conditioned Offline Reinforcement Learning through State Space Partitioningcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 2.

3 papers.

  1. 2025Retrosynthesis Planning via Worst-path Policy Optimisation in Tree-structured MDPscited by 2
  2. 2026Beyond Penalization: Diffusion-based Out-of-Distribution Detection and Selective Regularization in Offline Reinforcement Learningcited by 1
  3. 2026Unifying Value Alignment and Assignment in Cross-Domain Offline Reinforcement Learning with Heterogeneous Datasetscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  2. 2022A Policy-Guided Imitation Approach for Offline Reinforcement Learning
  3. 2021Mitigating Covariate Shift in Imitation Learning via Offline Data Without Great Coverage
  4. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  5. 2023Distance Weighted Supervised Learning for Offline Interaction Data
  6. 2022Discriminator-Weighted Offline Imitation Learning from Suboptimal Demonstrations
  7. 2023Offline Imitation Learning with Suboptimal Demonstrations via Relaxed Distribution Matching
  8. 2021Curriculum Offline Imitation Learning