mlsanityData to 14 Oct 2026

Offline Reinforcement Learning at Multiple Frequencies

arXiv 2207.13082 ↗First posted 26 Jul 2022cs.LG cs.AI cs.ROcited by 4

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Leveraging many sources of offline robot data requires grappling with the heterogeneity of such data. In this paper, we focus on one particular aspect of heterogeneity: learning from offline data collected at different control frequencies. Across labs, the discretization of controllers, sampling rates of sensors, and demands of a task of interest may differ, giving rise to a mixture of frequencies in an aggregated dataset. We study how well offline reinforcement learning (RL) algorithms can accommodate data with a mixture of frequencies during training. We observe that the Q-value propagates at different rates for different discretizations, leading to a number of learning challenges for off-the-shelf offline RL. We present a simple yet effective solution that enforces consistency in the rate of Q-value updates to stabilize learning. By scaling the value of N in N-step returns with the discretization size, we effectively balance Q-value propagation, leading to more stable convergence. On three simulated robotic control problems, we empirically find that this simple approach outperforms na"ive mixing by 50% on average.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 31 of its 46 references that are in the map. First posted on arXiv 26 Jul 2022.

Builds on

31 of 46 references.

Closest ancestors · 16

Same direction, newest first

  1. 2021You Only Evaluate Once: a Simple Baseline Algorithm for Offline RL5 Oct
  2. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning22 Sep
  3. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  4. 2021Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning17 May
  5. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  6. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  7. 2020Offline Reinforcement Learning from Images with Latent Space Models21 Dec
  8. 2020PLAS: Latent Action Space for Offline Reinforcement Learning14 Nov
  9. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning12 Aug
  10. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  11. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  12. 2020MOPO: Model-based Offline Policy Optimization27 May
4 more
  1. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  3. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  4. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 15

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016OpenAI Gymcited by 1,736
  3. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  4. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
  7. 2020HiPPO: Recurrent Memory with Optimal Polynomial Projectionscited by 605
  8. 2020Neural Controlled Differential Equations for Irregular Time Seriescited by 506
  9. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  10. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  11. 2019Making Deep Q-learning methods robust to time discretizationcited by 58
  12. 2020Neural Dynamic Policies for End-to-End Sensorimotor Learningcited by 58
3 more
  1. 2020Thinking While Moving: Deep Reinforcement Learning with Concurrent Controlcited by 31
  2. 2019Hamilton-Jacobi-Bellman Equations for Q-Learning in Continuous Timecited by 9
  3. 2022Multiscale Sensor Fusion and Continuous Control with Neural CDEscited by 4

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 1; 2024: 1.

4 papers.

  1. 2022Semi-Supervised Offline Reinforcement Learning with Action-Free Trajectoriescited by 28
  2. 2024When to Sense and Control? A Time-adaptive Approach for Continuous-Time RLcited by 9
  3. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learningcited by 8
  4. 2022Managing Temporal Resolution in Continuous Value Estimation: A Fundamental Trade-offcited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024On Sample-Efficient Offline Reinforcement Learning: Data Diversity, Posterior Sampling, and Beyond
  2. 2023Harnessing Mixed Offline Reinforcement Learning Datasets via Trajectory Weighting
  3. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets
  4. 2021Bridging Offline Reinforcement Learning and Imitation Learning: A Tale of Pessimism
  5. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data
  6. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble
  7. 2022Bridging the Gap Between Offline and Online Reinforcement Learning Evaluation Methodologies
  8. 2026Offline-Online Reinforcement Learning for Linear Mixture MDPs