# Integrating Domain Knowledge for handling Limited Data in Offline RL

arXiv: https://arxiv.org/abs/2406.07041 · HTML: https://mlsanity.com/paper/2406.07041/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 11 Jun 2024.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 27 of 27)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [OpenAI Gym](https://mlsanity.com/paper/1606.01540/) (2016)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Off-Policy Deep Reinforcement Learning without Exploration](https://mlsanity.com/paper/1812.02900/) (2018)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [Distributional Reinforcement Learning with Quantile Regression](https://mlsanity.com/paper/1710.10044/) (2017)
- [Minigrid & Miniworld: Modular & Customizable Reinforcement Learning Environments for Goal-Oriented Tasks](https://mlsanity.com/paper/2306.13831/) (2023)
- [Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble](https://mlsanity.com/paper/2110.01548/) (2021)
- [A Simple Semi-Supervised Learning Framework for Object Detection](https://mlsanity.com/paper/2005.04757/) (2020)
- [Critic Regularized Regression](https://mlsanity.com/paper/2006.15134/) (2020)
- [Offline Reinforcement Learning with Fisher Divergence Critic Regularization](https://mlsanity.com/paper/2103.08050/) (2021)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [A Theoretically Grounded Application of Dropout in Recurrent Neural Networks](https://mlsanity.com/paper/1512.05287/) (2015)
- [Distilling Task-Specific Knowledge from BERT into Simple Neural Networks](https://mlsanity.com/paper/1903.12136/) (2019)
- [Harnessing Deep Neural Networks with Logic Rules](https://mlsanity.com/paper/1603.06318/) (2016)
- [Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning](https://mlsanity.com/paper/2105.08140/) (2021)
- [Benchmarking Batch Deep Reinforcement Learning Algorithms](https://mlsanity.com/paper/1910.01708/) (2019)
- [S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning](https://mlsanity.com/paper/2103.06326/) (2021)
- [Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System](https://mlsanity.com/paper/1809.07428/) (2018)
- [Regularized Behavior Value Estimation](https://mlsanity.com/paper/2103.09575/) (2021)
- [A Dataset Perspective on Offline Reinforcement Learning](https://mlsanity.com/paper/2111.04714/) (2021)
- [Boosting Offline Reinforcement Learning with Action Preference Query](https://mlsanity.com/paper/2306.03362/) (2023)
- [Improving Offline RL by Blending Heuristics](https://mlsanity.com/paper/2306.00321/) (2023)

## Built on by (most cited first, 1 of 1)

- [Prompting Decision Transformers for Zero-Shot Reach-Avoid Policies](https://mlsanity.com/paper/2505.19337/) (2025)

## Abstract (arXiv)

With the ability to learn from static datasets, Offline Reinforcement Learning (RL) emerges as a compelling avenue for real-world applications. However, state-of-the-art offline RL algorithms perform sub-optimally when confronted with limited data confined to specific regions within the state space. The performance degradation is attributed to the inability of offline RL algorithms to learn appropriate actions for rare or unseen observations. This paper proposes a novel domain knowledge-based regularization technique and adaptively refines the initial domain knowledge to considerably boost performance in limited data with partially omitted states. The key insight is that the regularization term mitigates erroneous actions for sparse samples and unobserved states covered by domain knowledge. Empirical evaluations on standard discrete environment datasets demonstrate a substantial average performance increase of at least 27% compared to existing offline RL algorithms operating on limited data.