# Dataset Distillation for Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2407.20299 · HTML: https://mlsanity.com/paper/2407.20299/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 23 of its 31 references that are in the map.
- First posted on arXiv 29 Jul 2024.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [TinyBERT: Distilling BERT for Natural Language Understanding](https://mlsanity.com/paper/1909.10351/) (2019)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)
- [Distillation as a Defense to Adversarial Perturbations against Deep Neural Networks](https://mlsanity.com/paper/1511.04508/) (2015)
- [Patient Knowledge Distillation for BERT Model Compression](https://mlsanity.com/paper/1908.09355/) (2019)
- [A Brief Survey of Deep Reinforcement Learning](https://mlsanity.com/paper/1708.05866/) (2017)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Dataset Distillation by Matching Training Trajectories](https://mlsanity.com/paper/2203.11932/) (2022)
- [Policy Distillation](https://mlsanity.com/paper/1511.06295/) (2015)
- [A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems](https://mlsanity.com/paper/2203.01387/) (2022)
- [Does Knowledge Distillation Really Work?](https://mlsanity.com/paper/2106.05945/) (2021)
- [Distilling Task-Specific Knowledge from BERT into Simple Neural Networks](https://mlsanity.com/paper/1903.12136/) (2019)
- [Dataset Distillation: A Comprehensive Review](https://mlsanity.com/paper/2301.07014/) (2023)
- [A Comprehensive Survey of Dataset Distillation](https://mlsanity.com/paper/2301.05603/) (2023)
- [Zero-Shot Task Generalization with Multi-Task Deep Reinforcement Learning](https://mlsanity.com/paper/1706.05064/) (2017)
- [Distilling Policy Distillation](https://mlsanity.com/paper/1902.02186/) (2019)
- [Reinforced Multi-Teacher Selection for Knowledge Distillation](https://mlsanity.com/paper/2012.06048/) (2020)
- [ChainerRL: A Deep Reinforcement Learning Library](https://mlsanity.com/paper/1912.03905/) (2019)
- [Private Model Compression via Knowledge Distillation](https://mlsanity.com/paper/1811.05072/) (2018)

## Built on by (most cited first, 5 of 5)

- [Provable Domain Adaptation for Offline Reinforcement Learning with Limited Samples](https://mlsanity.com/paper/2408.12136/) (2024)
- [Analytic Energy-Guided Policy Optimization for Offline Reinforcement Learning](https://mlsanity.com/paper/2505.01822/) (2025)
- [Distilling Reinforcement Learning into Single-Batch Datasets](https://mlsanity.com/paper/2508.09283/) (2025)
- [Decision Flow Policy Optimization](https://mlsanity.com/paper/2505.20350/) (2025)
- [Algorithmic Guarantees for Distilling Supervised and Offline RL Datasets](https://mlsanity.com/paper/2512.00536/) (2025)

## Abstract (arXiv)

Offline reinforcement learning often requires a quality dataset that we can train a policy on. However, in many situations, it is not possible to get such a dataset, nor is it easy to train a policy to perform well in the actual environment given the offline data. We propose using data distillation to train and distill a better dataset which can then be used for training a better policy model. We show that our method is able to synthesize a dataset where a model trained on it achieves similar performance to a model trained on the full dataset or a model trained using percentile behavioral cloning. Our project site is available at https://datasetdistillation4rl.github.io . We also provide our implementation at https://github.com/ggflow123/DDRL .