Offline RL With Resource Constrained Online Deployment
On the map: Offline RL and imitation learning from data → Offline RL algorithms
Abstract (arXiv)
Offline reinforcement learning is used to train policies in scenarios where real-time access to the environment is expensive or impossible. As a natural consequence of these harsh conditions, an agent may lack the resources to fully observe the online environment before taking an action. We dub this situation the resource-constrained setting. This leads to situations where the offline dataset (available for training) can contain fully processed features (using powerful language models, image models, complex sensors, etc.) which are not available when actions are actually taken online. This disconnect leads to an interesting and unexplored problem in offline RL: Is it possible to use a richly processed offline dataset to train a policy which has access to fewer features in the online environment? In this work, we introduce and formalize this novel resource-constrained problem setting. We highlight the performance gap between policies trained using the full offline dataset and policies trained using limited features. We address this performance gap with a policy transfer algorithm which first trains a teacher agent using the offline dataset where features are fully available, and then transfers this knowledge to a student agent that only uses the resource-constrained features. To better capture the challenge of this setting, we propose a data collection procedure: Resource Constrained-Datasets for RL (RC-D4RL). We evaluate our transfer algorithm on RC-D4RL and the popular D4RL benchmarks and observe consistent improvement over the baseline (TD3+BC without transfer). The code for the experiments is available at https://github.com/JayanthRR/RC-OfflineRL.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 36 of its 57 references that are in the map. First posted on arXiv 7 Oct 2021.
Builds on
36 of 57 references.
Closest ancestors · 12
Same direction, newest first
- 2021Behavioral Priors and Dynamics Models: Improving Performance and Domain Transfer in Offline RL16 Jun
- 2021Offline RL Without Off-Policy Evaluation16 Jun
- 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
- 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
- 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
- 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
- 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
- 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
- 2020MOReL : Model-Based Offline Reinforcement Learning12 May
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
- 2019Behavior Regularized Offline Reinforcement Learning26 Nov
Other references · 24
- 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
- 2015Continuous control with deep reinforcement learningcited by 3,407
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,274
- 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
- 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,258
- 2020Knowledge Distillation: A Surveycited by 1,603
- 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
- 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
- 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
- 2017Deep Reinforcement Learning that Matterscited by 754
- 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
- 2015Policy Distillationcited by 440
12 more
- 2020Knowledge Distillation and Student-Teacher Learning for Visual Intelligence: A Review and New Outlookscited by 316
- 2020Transfer Learning in Deep Reinforcement Learning: A Surveycited by 265
- 2019Batch Policy Learning under Constraintscited by 244
- 2017Domain Generalization by Marginal Transfer Learningcited by 180
- 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
- 2017Learning Invariant Feature Spaces to Transfer Skills with Reinforcement Learningcited by 85
- 2019Distilling Policy Distillationcited by 84
- 2019A Generalization Error Bound for Multi-class Domain Generalizationcited by 26
- 2019Continual Reinforcement Learning deployed in Real-life using Policy Distillation and Sim2Real Transfercited by 26
- 2018Simple Regret Minimization for Contextual Banditscited by 16
- 2019Cross-View Policy Learning for Street Navigationcited by 12
- 2020Privileged Information Dropout in Reinforcement Learningcited by 8
Built on by
Citing papers per year: 2023: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
- 2022Leveraging Offline Data in Online Reinforcement Learning
- 2020Model-Based Offline Planning
- 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
- 2021The Least Restriction for Offline Reinforcement Learning
- 2020MOPO: Model-based Offline Policy Optimization
- 2025Offline vs. Online Learning in Model-based RL: Lessons for Data Collection Strategies
- 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems