mlsanityData to 14 Oct 2026

Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning

arXiv 2306.15503 ↗First posted 27 Jun 2023cs.LG cs.AIcited by 3

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

In recent years, data-driven reinforcement learning (RL), also known as offline RL, have gained significant attention. However, the role of data sampling techniques in offline RL has been overlooked despite its potential to enhance online RL performance. Recent research suggests applying sampling techniques directly to state-transitions does not consistently improve performance in offline RL. Therefore, in this study, we propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories for more comprehensive information extraction from limited data. TR enhances learning efficiency by backward sampling of trajectories that optimizes the use of subsequent state information. Building on TR, we build the weighted critic target to avoid sampling unseen actions in offline training, and Prioritized Trajectory Replay (PTR) that enables more efficient trajectory sampling, prioritized by various trajectory priority metrics. We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL. In summary, our research emphasizes the significance of trajectory-based data sampling techniques in enhancing the efficiency and performance of offline RL algorithms.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 28 of its 37 references that are in the map. First posted on arXiv 27 Jun 2023.

Builds on

28 of 37 references.

Closest ancestors · 12

Same direction, newest first

  1. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  2. 2022Boosting Offline Reinforcement Learning via Data Rebalancing17 Oct
  3. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
  4. 2022Pessimistic Bootstrapping for Uncertainty-Driven Offline Reinforcement Learning23 Feb
  5. 2021Dealing with the Unknown: Pessimistic Offline Reinforcement Learning9 Nov
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Uncertainty-Based Offline Reinforcement Learning with Diversified Q-Ensemble4 Oct
  8. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  9. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  10. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  11. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  12. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 16

  1. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  2. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  3. 2015Prioritized Experience Replaycited by 1,099
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2020Deep Reinforcement Learning for Autonomous Driving: A Surveycited by 721
  7. 2020Reinforcement Learning with Augmented Datacited by 468
  8. 2019Mastering Complex Control in MOBA Games with Deep Reinforcement Learningcited by 161
  9. 2018Self-Imitation Learningcited by 138
  10. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 58
  11. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  12. 2018Sample-Efficient Deep Reinforcement Learning via Episodic Backward Updatecited by 29
4 more
  1. 2020Understanding and Mitigating the Limitations of Prioritized Experience Replaycited by 17
  2. 2022Topological Experience Replaycited by 17
  3. 2021Revisiting Prioritized Experience Replay: A Value Perspectivecited by 7
  4. 2022PAnDR: Fast Adaptation to New Environments from Offline Experiences via Decoupling Policy and Environment Representationscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 1.

3 papers.

  1. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  2. 2026Continual Policy Consolidation for Lifelong Robot Learningcited by 3
  3. 2025Towards Interpretable and Inference-Optimal COT Reasoning with Sparse Autoencoder-Guided Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  2. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  3. 2023Decoupled Prioritized Resampling for Offline RL
  4. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  5. 2024TrajDeleter: Enabling Trajectory Forgetting in Offline Reinforcement Learning Agents
  6. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  7. 2025Adaptive Replay Buffer for Offline-to-Online Reinforcement Learning
  8. 2021Representation Matters: Offline Pretraining for Sequential Decision Making