mlsanityData to 14 Oct 2026

Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning

arXiv 2303.05479 ↗First posted 9 Mar 2023cs.LG cs.AIcited by 237

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

A compelling use case of offline reinforcement learning (RL) is to obtain a policy initialization from existing datasets followed by fast online fine-tuning with limited interaction. However, existing offline RL methods tend to behave poorly during fine-tuning. In this paper, we devise an approach for learning an effective initialization from offline data that also enables fast online fine-tuning capabilities. Our approach, calibrated Q-learning (Cal-QL), accomplishes this by learning a conservative value function initialization that underestimates the value of the learned policy from offline data, while also being calibrated, in the sense that the learned Q-values are at a reasonable scale. We refer to this property as calibration, and define it formally as providing a lower bound on the true value function of the learned policy and an upper bound on the value of some other (suboptimal) reference policy, which may simply be the behavior policy. We show that offline RL algorithms that learn such calibrated value functions lead to effective online fine-tuning, enabling us to take the benefits of offline initializations in online fine-tuning. In practice, Cal-QL can be implemented on top of the conservative Q learning (CQL) for offline RL within a one-line code change. Empirically, Cal-QL outperforms state-of-the-art methods on 9/11 fine-tuning benchmark tasks that we study in this paper. Code and video are available at https://nakamotoo.github.io/Cal-QL

Key facts. Cited by 237 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 49 of its 65 references that are in the map. First posted on arXiv 9 Mar 2023.

Builds on

49 of 65 references.

Closest ancestors · 21

Same direction, newest first

  1. 2023The In-Sample Softmax for Offline Reinforcement Learning28 Feb
  2. 2022Leveraging Offline Data in Online Reinforcement Learning9 Nov
  3. 2022Mildly Conservative Q-Learning for Offline Reinforcement Learning9 Jun
  4. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  5. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning19 Aug
  8. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  9. 2021Bellman-consistent Pessimism for Offline Reinforcement Learning13 Jun
  10. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  11. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learning9 Jun
  12. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
9 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?30 Dec
  2. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learning26 Oct
  3. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RL21 Jul
  4. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  5. 2020MOPO: Model-based Offline Policy Optimization27 May
  6. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learning19 Feb

Other references · 28

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2015Continuous control with deep reinforcement learningcited by 3,407
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  7. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,240
  8. 2017Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrationscited by 715
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  10. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  11. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  12. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
16 more
  1. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  2. 2021Bellman Eluder Dimension: New Rich Classes of RL Problems, and Sample-Efficient Algorithmscited by 223
  3. 2022Online Decision Transformercited by 196
  4. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  5. 2021Bilinear Classes: A Structural Framework for Provable Generalization in RLcited by 178
  6. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  7. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  8. 2020The Importance of Pessimism in Fixed-Dataset Policy Optimizationcited by 99
  9. 2018Dexterous Manipulation with Deep Reinforcement Learning: Efficient, General, and Low-Costcited by 84
  10. 2020COG: Connecting New Skills to Past Experience with Offline Reinforcement Learningcited by 83
  11. 2020Towards Playing Full MOBA Games with Deep Reinforcement Learningcited by 78
  12. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  13. 2022Unpacking Reward Shaping: Understanding the Benefits of Reward Engineering on Sample Complexitycited by 62
  14. 2022Improving TD3-BC: Relaxed Policy Constraint for Offline Learning and Stable Online Fine-Tuningcited by 20
  15. 2021Computational Benefits of Intermediate Rewards for Goal-Reaching Policy Learningcited by 17
  16. 2022Understanding the Complexity Gains of Single-Task RL with a Curriculumcited by 17

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 30; 2024: 48; 2025: 67; 2026: 92.

237 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  2. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
  3. 2024Diffusion Policy Policy Optimizationcited by 280
  4. 2025Flow Q-Learningcited by 184
  5. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  6. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 154
  7. 2024THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulationcited by 133
  8. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
  9. 2025Reinforcement Learning with Action Chunkingcited by 107
  10. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 106
  11. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  12. 2024Foundation Policies with Hilbert Representationscited by 94
38 more
  1. 2024From Imitation to Refinement -- Residual RL for Precise Assemblycited by 90
  2. 2023LMRL Gym: Benchmarks for Multi-Turn Reinforcement Learning with Language Modelscited by 88
  3. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 79
  4. 2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
  5. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  6. 2024Efficient Online Reinforcement Learning Fine-Tuning Need Not Retain Offline Datacited by 63
  7. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  8. 2025Hume: Introducing System-2 Thinking in Visual-Language-Action Modelcited by 56
  9. 2023Robot Fine-Tuning Made Easy: Pre-Training Rewards and Policies for Autonomous Real-World Reinforcement Learningcited by 54
  10. 2024FLaRe: Achieving Masterful and Adaptive Robot Policies with Large-Scale Reinforcement Learning Fine-Tuningcited by 54
  11. 2024Cal-DPO: Calibrated Direct Preference Optimization for Language Model Alignmentcited by 51
  12. 2024DiffuseLoco: Real-Time Legged Locomotion Control with Diffusion from Offline Datasetscited by 49
  13. 2026RL Token: Bootstrapping Online RL with Vision-Language-Action Modelscited by 49
  14. 2023Finetuning Offline World Models in the Real Worldcited by 41
  15. 2025CO-RFT: Efficient Fine-Tuning of Vision-Language-Action Models through Chunked Offline Reinforcement Learningcited by 37
  16. 2026Q-learning with Adjoint Matchingcited by 37
  17. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  18. 2025A Survey on Efficient Vision-Language-Action Modelscited by 34
  19. 2026TOPReward: Token Probabilities as Hidden Zero-Shot Rewards for Roboticscited by 33
  20. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  21. 2023RLIF: Interactive Imitation Learning as Reinforcement Learningcited by 30
  22. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learningcited by 29
  23. 2024ASID: Active Exploration for System Identification in Robotic Manipulationcited by 28
  24. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  25. 2023Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimizationcited by 26
  26. 2024Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problemcited by 23
  27. 2024Affordance-Guided Reinforcement Learning via Visual Promptingcited by 23
  28. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  29. 2023Accelerating Exploration with Unlabeled Prior Datacited by 21
  30. 2025Fast Adaptation with Behavioral Foundation Modelscited by 21
  31. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensemblescited by 20
  32. 2023H2O+: An Improved Framework for Hybrid Offline-and-Online RL with Dynamics Gapscited by 17
  33. 2024Energy-Guided Diffusion Sampling for Offline-to-Online Reinforcement Learningcited by 17
  34. 2025Rapidly Adapting Policies to the Real World via Simulation-Guided Fine-Tuningcited by 17
  35. 2023Reward-agnostic Fine-tuning: Provable Statistical Benefits of Hybrid Reinforcement Learningcited by 16
  36. 2025What Matters for Batch Online Reinforcement Learning in Robotics?cited by 15
  37. 2025Value Flowscited by 15
  38. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 15

50 most cited of 237 All 237 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Online RL Fine Tuning with Offline Pre-trained Policy Only
  2. 2023A Perspective of Q-value Estimation on Offline-to-Online Reinforcement Learning
  3. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  4. 2023Offline Reinforcement Learning with On-Policy Q-Function Regularization
  5. 2024Exclusively Penalized Q-learning for Offline Reinforcement Learning
  6. 2021Offline RL Without Off-Policy Evaluation
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions