mlsanityData to 7 Oct 2026

Conservative Data Sharing for Multi-Task Offline Reinforcement Learning

arXiv 2109.08128 ↗First posted 16 Sep 2021cs.LG cs.AI cs.ROcited by 64

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Offline reinforcement learning (RL) algorithms have shown promising results in domains where abundant pre-collected data is available. However, prior methods focus on solving individual problems from scratch with an offline dataset without considering how an offline RL agent can acquire multiple skills. We argue that a natural use case of offline RL is in settings where we can pool large amounts of data collected in various scenarios for solving different tasks, and utilize all of this data to learn behaviors for all the tasks more effectively rather than training each one in isolation. However, sharing data across all tasks in multi-task offline RL performs surprisingly poorly in practice. Thorough empirical analysis, we find that sharing data can actually exacerbate the distributional shift between the learned policy and the dataset, which in turn can lead to divergence of the learned policy and poor performance. To address this challenge, we develop a simple technique for data-sharing in multi-task offline RL that routes data based on the improvement over the task-specific data. We call this approach conservative data sharing (CDS), and it can be applied with multiple single-task offline RL methods. On a range of challenging multi-task locomotion, navigation, and vision-based robotic manipulation problems, CDS achieves the best or comparable performance compared to prior offline multi-task RL methods and previous data sharing approaches.

Key facts. Cited by 64 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 69 of its 92 references that are in the map. First posted on arXiv 16 Sep 2021.

Builds on

69 of 92 references.

Closest ancestors · 15

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  2. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  3. 2021Representation Matters: Offline Pretraining for Sequential Decision Making11 Feb
  4. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  5. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  6. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  7. 2020Provably Good Batch Reinforcement Learning Without Great Exploration16 Jul
  8. 2020RL Unplugged: A Suite of Benchmarks for Offline Reinforcement Learning24 Jun
  9. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  10. 2020MOPO: Model-based Offline Policy Optimization27 May
  11. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  12. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
3 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  2. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb
  3. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 54

  1. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  2. 2017Deep reinforcement learning from human preferencescited by 3,846
  3. 2015Continuous control with deep reinforcement learningcited by 3,279
  4. 2015Trust Region Policy Optimizationcited by 2,503
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2017Hindsight Experience Replaycited by 1,096
  10. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  11. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  12. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
38 more
  1. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  2. 2015Policy Distillationcited by 414
  3. 2018Maximum a Posteriori Policy Optimisationcited by 318
  4. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  5. 2016Deep Visual Foresight for Planning Robot Motioncited by 276
  6. 2019RoboNet: Large-Scale Multi-Robot Learningcited by 268
  7. 2018Visual Reinforcement Learning with Imagined Goalscited by 248
  8. 2017Distral: Robust Multitask Reinforcement Learningcited by 231
  9. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  10. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  11. 2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 176
  12. 2021Multi-Task Reinforcement Learning with Context-based Representationscited by 169
  13. 2017Safe Policy Improvement with Baseline Bootstrappingcited by 141
  14. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  15. 2020Multi-Task Reinforcement Learning with Soft Modularizationcited by 138
  16. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  17. 2020Model-Based Offline Planningcited by 126
  18. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  19. 2020Deployment-Efficient Reinforcement Learning via Model-Based Offline Optimizationcited by 120
  20. 2021Learning Generalizable Robotic Reward Functions from "In-The-Wild" Human Videoscited by 116
  21. 2015An Emphatic Approach to the Problem of Off-policy Temporal-Difference Learningcited by 103
  22. 2018Supervised Reinforcement Learning with Recurrent Neural Network for Dynamic Treatment Recommendationcited by 101
  23. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  24. 2018Temporal Difference Models: Model-Free Deep RL for Model-Based Controlcited by 89
  25. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 85
  26. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  27. 2018Multiple Interactions Made Easy (MIME): Large Scale Demonstrations Data for Imitationcited by 76
  28. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  29. 2018Variational Inverse Control with Events: A General Framework for Data-Driven Reward Definitioncited by 66
  30. 2020Rewriting History with Inverse RL: Hindsight Inference for Policy Improvementcited by 59
  31. 2020Generalized Hindsight for Reinforcement Learningcited by 51
  32. 2017Divide-and-Conquer Reinforcement Learningcited by 46
  33. 2019Mapping State Space using Landmarks for Universal Goal Reachingcited by 44
  34. 2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
  35. 2019Improvisation through Physical Understanding: Using Novel Objects as Tools with Visual Foresightcited by 34
  36. 2020An Empirical Study of Representation Learning for Reinforcement Learning in Healthcarecited by 34
  37. 2018Few-Shot Goal Inference for Visuomotor Learning and Planningcited by 30
  38. 2020Knowledge Transfer in Multi-Task Deep Reinforcement Learning for Continuous Controlcited by 27

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 16; 2023: 16; 2024: 21; 2025: 8; 2026: 2.

64 papers.

Sort:

By area:

  1. 2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 564
  2. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 114
  3. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  4. 2022Pessimistic Q-Learning for Offline Reinforcement Learning: Towards Optimal Sample Complexitycited by 84
  5. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  6. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learningcited by 61
  7. 2022The Efficacy of Pessimism in Asynchronous Q-Learningcited by 41
  8. 2022Planning to Practice: Efficient Online Fine-Tuning by Composing Goals in Latent Spacecited by 31
  9. 2023Cross-Domain Policy Adaptation via Value-Guided Data Filteringcited by 30
  10. 2022Don't Start From Scratch: Leveraging Prior Data to Automate Robotic Reinforcement Learningcited by 28
  11. 2023Future-conditioned Unsupervised Pretraining for Decision Transformercited by 26
  12. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasetscited by 24
38 more
  1. 2022Multi-Task Imitation Learning for Linear Dynamical Systemscited by 22
  2. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
  3. 2022Using Both Demonstrations and Language Instructions to Efficiently Learn Robotic Taskscited by 17
  4. 2022Phasic Self-Imitative Reduction for Sparse-Reward Goal-Conditioned Reinforcement Learningcited by 16
  5. 2022Hierarchical Decision Transformercited by 16
  6. 2023Mind the Gap: Offline Policy Optimization for Imperfect Rewardscited by 15
  7. 2022Model-Based Offline Meta-Reinforcement Learning with Regularizationcited by 14
  8. 2024Offline Multitask Representation Learning for Reinforcement Learningcited by 14
  9. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  10. 2023Unsupervised Behavior Extraction via Random Intent Priorscited by 13
  11. 2024MOTO: Offline Pre-training to Online Fine-tuning for Model-based Robot Learningcited by 13
  12. 2024Traversing Pareto Optimal Policies: Provably Efficient Multi-Objective Reinforcement Learningcited by 13
  13. 2022Latent-Variable Advantage-Weighted Policy Optimization for Offline RLcited by 12
  14. 2023A Survey of Demonstration Learningcited by 12
  15. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  16. 2023The Provable Benefits of Unsupervised Data Sharing for Offline Reinforcement Learningcited by 10
  17. 2024Hokoff: Real Game Dataset from Honor of Kings and its Offline Reinforcement Learning Benchmarkscited by 10
  18. 2024Active Fine-Tuning of Multi-Task Policiescited by 8
  19. 2023QMP: Q-switch Mixture of Policies for Multi-Task Behavior Sharingcited by 6
  20. 2023Federated Ensemble-Directed Offline Reinforcement Learningcited by 6
  21. 2023On-Policy Policy Gradient Reinforcement Learning Without On-Policy Samplingcited by 6
  22. 2024Pessimistic Value Iteration for Multi-Task Data Sharing in Offline Reinforcement Learningcited by 6
  23. 2024Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learningcited by 6
  24. 2022Forgetting and Imbalance in Robot Lifelong Learning with Off-policy Datacited by 5
  25. 2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 5
  26. 2024Context in Public Health for Underserved Communities: A Bayesian Approach to Online Restless Banditscited by 5
  27. 2024A Behavior-Aware Approach for Deep Reinforcement Learning in Non-stationary Environments without Known Change Pointscited by 5
  28. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learningcited by 5
  29. 2023OER: Offline Experience Replay for Continual Offline Reinforcement Learningcited by 3
  30. 2023Deep Reinforcement Learning with Explicit Context Representationcited by 3
  31. 2024Skills Regularized Task Decomposition for Multi-task Offline Reinforcement Learningcited by 3
  32. 2025Efficient Multi-Task Reinforcement Learning with Cross-Task Policy Guidancecited by 3
  33. 2024Disentangling Policy from Offline Task Representation Learning via Adversarial Data Augmentationcited by 2
  34. 2024Urban-Focused Multi-Task Offline Reinforcement Learning with Contrastive Data Sharingcited by 2
  35. 2024Leveraging Unlabeled Data Sharing through Kernel Function Approximation in Offline Reinforcement Learningcited by 2
  36. 2024Offline Behavior Distillationcited by 2
  37. 2022A Simple Structure For Building A Robust Modelcited by 1
  38. 2024Efficient Multi-Task Reinforcement Learning via Task-Specific Action Correctioncited by 1

50 most cited of 64 All 64 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Leveraging Offline Data in Online Reinforcement Learning
  2. 2025Unsupervised Data Generation for Offline Reinforcement Learning: A Perspective from Model
  3. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  5. 2024Offline Multi-task Transfer RL with Representational Penalization
  6. 2024Simple Ingredients for Offline Reinforcement Learning
  7. 2025Evaluation-Time Policy Switching for Offline Reinforcement Learning
  8. 2020Offline Reinforcement Learning Hands-On