mlsanityData to 14 Oct 2026

Small Dataset, Big Gains: Enhancing Reinforcement Learning by Offline Pre-Training with Model Based Augmentation

arXiv 2312.09844 ↗First posted 15 Dec 2023cs.LG cs.AIcited by 6

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning leverages pre-collected datasets of transitions to train policies. It can serve as effective initialization for online algorithms, enhancing sample efficiency and speeding up convergence. However, when such datasets are limited in size and quality, offline pre-training can produce sub-optimal policies and lead to degraded online reinforcement learning performance. In this paper we propose a model-based data augmentation strategy to maximize the benefits of offline reinforcement learning pre-training and reduce the scale of data needed to be effective. Our approach leverages a world model of the environment trained on the offline dataset to augment states during offline pre-training. We evaluate our approach on a variety of MuJoCo robotic tasks and our results show it can jump-start online fine-tuning and substantially reduce - in some cases by an order of magnitude - the required number of environment interactions.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 14 of its 16 references that are in the map. First posted on arXiv 15 Dec 2023.

Builds on

14 of 16 references.

Closest ancestors · 6

Same direction, newest first

  1. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions30 Mar
  2. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 8

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,653
  4. 2018World Modelscited by 1,513
  5. 2020Mastering Atari with Discrete World Modelscited by 1,007
  6. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  7. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  8. 2022DayDreamer: World Models for Physical Robot Learningcited by 400

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 2.

6 papers.

  1. 2024Deep Reinforcement Learning in Autonomous Car Path Planning and Control: A Surveycited by 5
  2. 2024Is FISHER All You Need in The Multi-AUV Underwater Target Tracking Task?cited by 3
  3. 2025Edge General Intelligence Through World Models and Agentic AI: Fundamentals, Solutions, and Challengescited by 3
  4. 2025Human-Like Goalkeeping in a Realistic Football Simulation: a Sample-Efficient Reinforcement Learning Approachcited by 2
  5. 2024A Benchmark Environment for Offline Reinforcement Learning in Racing Gamescited by 1
  6. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimizationcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2023Finetuning Offline World Models in the Real World
  2. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  3. 2024Augmenting Offline Reinforcement Learning with State-only Interactions
  4. 2023Guided Data Augmentation for Offline Reinforcement Learning and Imitation Learning
  5. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  6. 2020MOPO: Model-based Offline Policy Optimization
  7. 2021Online and Offline Reinforcement Learning by Planning with a Learned Model
  8. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learning