mlsanityData to 7 Oct 2026

Uni-O4: Unifying Online and Offline Deep Reinforcement Learning with Multi-Step On-Policy Optimization

arXiv 2311.03351 ↗First posted 6 Nov 2023cs.LG cs.ROcited by 23

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Combining offline and online reinforcement learning (RL) is crucial for efficient and safe learning. However, previous approaches treat offline and online learning as separate procedures, resulting in redundant designs and limited performance. We ask: Can we achieve straightforward yet effective offline and online learning without introducing extra conservatism or regularization? In this study, we propose Uni-o4, which utilizes an on-policy objective for both offline and online learning. Owning to the alignment of objectives in two phases, the RL agent can transfer between offline and online learning seamlessly. This property enhances the flexibility of the learning paradigm, allowing for arbitrary combinations of pretraining, fine-tuning, offline, and online learning. In the offline phase, specifically, Uni-o4 leverages diverse ensemble policies to address the mismatch issues between the estimated behavior policy and the offline dataset. Through a simple offline policy evaluation (OPE) approach, Uni-o4 can achieve multi-step policy improvement safely. We demonstrate that by employing the method above, the fusion of these two paradigms can yield superior offline initialization as well as stable and rapid online fine-tuning capabilities. Through real-world robot tasks, we highlight the benefits of this paradigm for rapid deployment in challenging, previously unseen real-world environments. Additionally, through comprehensive evaluations using numerous simulated benchmarks, we substantiate that our method achieves state-of-the-art performance in both offline and offline-to-online fine-tuning learning. Our website: https://lei-kun.github.io/uni-o4/ .

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 55 of its 61 references that are in the map. First posted on arXiv 6 Nov 2023.

Builds on

55 of 61 references.

Closest ancestors · 25

Same direction, newest first

  1. 2023Beyond Uniform Sampling: Offline Reinforcement Learning with Imbalanced Datasets6 Oct
  2. 2023A Simple Unified Uncertainty-Guided Framework for Offline-to-Online Reinforcement Learning13 Jun
  3. 2023ENOTO: Improving Offline-to-Online Reinforcement Learning with Q-Ensembles12 Jun
  4. 2023PROTO: Iterative Policy Regularized Offline-to-Online Reinforcement Learning25 May
  5. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularization28 Mar
  6. 2023Adaptive Policy Learning for Offline-to-Online Reinforcement Learning14 Mar
  7. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuning9 Mar
  8. 2023Behavior Proximal Policy Optimization22 Feb
  9. 2023Policy Expansion for Bridging Offline-to-Online Reinforcement Learning2 Feb
  10. 2023Extreme Q-Learning: MaxEnt RL without Entropy5 Jan
  11. 2022Confidence-Conditioned Value Functions for Offline Reinforcement Learning8 Dec
  12. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Library13 Oct
13 more
  1. 2022When to Trust Your Simulator: Dynamics-Aware Hybrid Offline-and-Online Reinforcement Learning27 Jun
  2. 2022RORL: Robust Offline Reinforcement Learning via Conservative Smoothing6 Jun
  3. 2022Supported Policy Optimization for Offline Reinforcement Learning13 Feb
  4. 2022Adversarially Trained Actor Critic for Offline Reinforcement Learning5 Feb
  5. 2021RvS: What is Essential for Offline RL via Supervised Learning?20 Dec
  6. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  7. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemble1 Jul
  8. 2021Offline RL Without Off-Policy Evaluation16 Jun
  9. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  10. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  11. 2020Hyperparameter Selection for Offline Reinforcement Learning17 Jul
  12. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  13. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr

Other references · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  6. 2015Trust Region Policy Optimizationcited by 2,503
  7. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  9. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  12. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
18 more
  1. 2021Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learningcited by 395
  2. 2015Doubly Robust Off-policy Value Evaluation for Reinforcement Learningcited by 337
  3. 2023Efficient Online Reinforcement Learning with Offline Datacited by 332
  4. 2022Online Decision Transformercited by 194
  5. 2019Empirical Study of Off-Policy Policy Evaluation for Reinforcement Learningcited by 121
  6. 2022Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behaviorcited by 113
  7. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  8. 2022A Walk in the Park: Learning to Walk in 20 Minutes With Model-Free Reinforcement Learningcited by 76
  9. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  10. 2021Legged Robots that Keep on Learning: Fine-Tuning Locomotion Policies in the Real Worldcited by 64
  11. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  12. 2023SpawnNet: Learning Generalizable Visuomotor Skills from Pre-trained Networkscited by 24
  13. 2023Benchmarking Offline Reinforcement Learning on Real-Robot Hardwarecited by 23
  14. 2023Grow Your Limits: Continuous Improvement with Real-World RL for Robotic Locomotioncited by 21
  15. 2022Real World Offline Reinforcement Learning with Realistic Data Sourcecited by 16
  16. 2022Learning on the Job: Self-Rewarding Offline-to-Online Finetuning for Industrial Insertion of Novel Connectors from Visioncited by 11
  17. 2023Decision Transformer under Random Frame Droppingcited by 10
  18. 2023Manipulate by Seeing: Creating Manipulation Controllers from Pre-Trained Representationscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 2; 2024: 8; 2025: 6; 2026: 7.

23 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  3. 20243D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representationscited by 115
  4. 2023Diffusion Reward: Learning Rewards via Conditional Video Diffusioncited by 56
  5. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 47
  6. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
  7. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learningcited by 33
  8. 2024Learning Visual Quadrupedal Loco-Manipulation from Demonstrationscited by 15
  9. 2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 12
  10. 2024Unsupervised-to-Online Reinforcement Learningcited by 11
  11. 2024Optimistic Critic Reconstruction and Constrained Fine-Tuning for General Offline-to-Online RLcited by 9
  12. 2025Penalizing Infeasible Actions and Reward Scaling in Reinforcement Learning with Offline Datacited by 7
11 more
  1. 2026Failure-Aware RL: Reliable Offline-to-Online Reinforcement Learning with Self-Recovery for Real-World Manipulationcited by 6
  2. 2023Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimizationcited by 3
  3. 2024Offline Reinforcement Learning with Behavioral Supervisor Tuningcited by 3
  4. 2024Offline-to-Online Multi-Agent Reinforcement Learning with Offline Value Function Memory and Sequential Explorationcited by 3
  5. 2026ViTaS: Visual Tactile Soft Fusion Contrastive Learning for Visuomotor Learningcited by 3
  6. 2026Trust Region Q Adjoint Matchingcited by 3
  7. 2026SOPE: Stabilizing Off-Policy Evaluation for Online RL with Prior Datacited by 1
  8. 2025Learning from Suboptimal Data in Continuous Control via Auto-Regressive Soft Q-Networkcited by 0
  9. 2025Online Pre-Training for Offline-to-Online Reinforcement Learningcited by 0
  10. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learningcited by 0
  11. 2026COOPO: Cyclic Offline-Online Policy Optimization Algorithmcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Active Offline-to-Online Reinforcement Learning
  2. 2024Offline-Boosted Actor-Critic: Adaptively Blending Optimal Historical Behaviors in Deep Off-Policy RL
  3. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problems
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2025A Clean Slate for Offline Reinforcement Learning
  6. 2023Beyond OOD State Actions: Supported Cross-Domain Offline Reinforcement Learning
  7. 2023Planning to Go Out-of-Distribution in Offline-to-Online Reinforcement Learning
  8. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models