mlsanityData to 7 Oct 2026

Integrating Domain Knowledge for handling Limited Data in Offline RL

arXiv 2406.07041 ↗First posted 11 Jun 2024cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

With the ability to learn from static datasets, Offline Reinforcement Learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-the-art offline RL algorithms perform sub-optimally when confronted with limited data confined to specific regions within the state space. The performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This paper proposes a novel domain knowledge-based regularization technique and adaptively refines the initial domain knowledge to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase of at least 27% compared to existing offline RL algorithms operating on limited data.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 27 of its 45 references that are in the map. First posted on arXiv 11 Jun 2024.

Builds on

27 of 45 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Boosting Offline Reinforcement Learning with Action Preference Query6 Jun
  2. 2023Improving Offline RL by Blending Heuristics1 Jun
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems2 Mar
  4. 2021A Dataset Perspective on Offline Reinforcement Learning8 Nov
  5. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  6. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  7. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  8. 2021Regularized Behavior Value Estimation17 Mar
  9. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  10. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning10 Mar
  11. 2020Critic Regularized Regression26 Jun
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
3 more
  1. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  3. 2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 12

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  2. 2016OpenAI Gymcited by 1,684
  3. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  4. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  7. 2023Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Taskscited by 320
  8. 2020A Simple Semi-Supervised Learning Framework for Object Detectioncited by 282
  9. 2015A Theoretically Grounded Application of Dropout in Recurrent Neural Networkscited by 161
  10. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  11. 2016Harnessing Deep Neural Networks with Logic Rulescited by 135
  12. 2018Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender Systemcited by 60

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Prompting Decision Transformers for Zero-Shot Reach-Avoid Policiescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Domain Generalization for Robust Model-Based Offline Reinforcement Learning
  2. 2024Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples
  3. 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL
  4. 2022Regularizing a Model-based Policy Stationary Distribution to Stabilize Offline Reinforcement Learning
  5. 2025Sparse-Reg: Improving Sample Complexity in Offline Reinforcement Learning using Sparsity
  6. 2022Leveraging Offline Data in Online Reinforcement Learning
  7. 2023Policy Regularization with Dataset Constraint for Offline Reinforcement Learning
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Making