Data to 14 Oct 2026

Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbone

arXiv 2412.06685 ↗First posted 9 Dec 2024cs.LG cs.AIcited by 79

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Recent advances in learning decision-making policies can largely be attributed to training expressive policy models, largely via imitation learning. While imitation learning discards non-expert data, reinforcement learning (RL) can still learn from suboptimal data. However, instantiating RL training of a new policy class often presents a different challenge: most deep RL machinery is co-developed with assumptions on the policy class and backbone, resulting in poor performance when the policy class changes. For instance, SAC utilizes a low-variance reparameterization policy gradient for Gaussian policies, but this is unstable for diffusion policies and intractable for autoregressive categorical policies. To address this issue, we develop an offline RL and online fine-tuning approach called policy-agnostic RL (PA-RL) that can effectively train multiple policy classes, with varying architectures and sizes. We build off the basic idea that a universal supervised learning loss can replace the policy improvement step in RL, as long as it is applied on "optimized" actions. To obtain these optimized actions, we first sample multiple actions from a base policy, and run global optimization (i.e., re-ranking multiple action samples using the Q-function) and local optimization (i.e., running gradient steps on an action sample) to maximize the critic on these candidates. PA-RL enables fine-tuning diffusion and transformer policies with either autoregressive tokens or continuous action outputs, at different sizes, entirely via actor-critic RL. Moreover, PA-RL improves the performance and sample-efficiency by up to 2 times compared to existing offline RL and online fine-tuning methods. We show the first result that successfully fine-tunes OpenVLA, a 7B generalist robot policy, autonomously with Cal-QL, an online RL fine-tuning algorithm, improving from 40% to 70% in the real world in 40 minutes.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 50 of its 59 references that are in the map. First posted on arXiv 9 Dec 2024.

Builds on

50 of 59 references.

Closest ancestors · 8

Same direction, newest first

Other references · 42

2020Denoising Diffusion Probabilistic Modelscited by 21,076
2017Proximal Policy Optimization Algorithmscited by 15,215
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
2015Continuous control with deep reinforcement learningcited by 3,407
2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,838
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 743
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671

30 more

2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 657
2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
2021CALVIN: A Benchmark for Language-Conditioned Policy Learning for Long-Horizon Robot Manipulation Taskscited by 584
2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 562
2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
2023Efficient Online Reinforcement Learning with Offline Datacited by 363
2018Maximum a Posteriori Policy Optimisationcited by 328
2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policiescited by 325
2024Diffusion Policy Policy Optimizationcited by 280
2023FireAct: Toward Language Agent Fine-tuningcited by 256
2021Bridge Data: Boosting Generalization of Robotic Skills with Cross-Domain Datasetscited by 250
2024DigiRL: Training In-The-Wild Device-Control Agents with Autonomous Reinforcement Learningcited by 214
2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
2018Self-Imitation Learningcited by 138
2023Learning a Diffusion Model Policy from Rewards via Q-Score Matchingcited by 138
2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 131
2023Policy Representation via Diffusion Probability Model for Reinforcement Learningcited by 112
2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 106
2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RLcited by 103
2019Reward-Conditioned Policiescited by 102
2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
2022Skill-based Model-based Reinforcement Learningcited by 50
2019Leveraging exploration in off-policy algorithms via normalizing flowscited by 46
2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
2020GRAC: Self-Guided and Self-Regularized Actor-Criticcited by 21
2019Q-Learning for Continuous Actions with Cross-Entropy Guided Policiescited by 18
2018Greedy Actor-Critic: A New Conditional Cross-Entropy Method for Policy Improvementcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 33; 2026: 46.

79 papers.

Sort:

By area:

2025π*0.6: a VLA That Learns From Experiencecited by 303
2025Flow Q-Learningcited by 184
2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 154
2025What Can RL Bring to VLA Generalization? An Empirical Studycited by 108
2025Self-Improving Vision-Language-Action Models with Data Generation via Residual RLcited by 78
2025VLA-RFT: Vision-Language-Action Reinforcement Fine-tuning with Verified Rewards in World Simulatorscited by 67
2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
2025Residual Off-Policy RL for Finetuning Behavior Cloning Policiescited by 55
2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 47
2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 40
2026Q-learning with Adjoint Matchingcited by 37

38 more

2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
2025DiWA: Diffusion Policy Adaptation with World Modelscited by 31
2025SAC Flow: Sample-Efficient Reinforcement Learning of Flow-Based Policies via Velocity-Reparameterized Sequential Modelingcited by 31
2025Compliant Residual DAgger: Improving Real-World Contact-Rich Manipulation with Human Correctionscited by 29
2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
2025Robust Finetuning of Vision-Language-Action Robot Policies via Parameter Mergingcited by 22
2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 18
2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
2025Balancing Signal and Variance: Adaptive Offline RL Post-Training for VLA Flow Modelscited by 15
2026On-the-Fly VLA Adaptation via Test-Time Reinforcement Learningcited by 14
2026EXPO-FT: Sample-Efficient Reinforcement Learning Finetuning for Vision-Language-Action Modelscited by 11
2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 8
2025Posterior Behavioral Cloning: Pretraining BC Policies for Efficient RL Finetuningcited by 8
2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
2025Score-Based Diffusion Policy Compatible with Reinforcement Learning via Optimal Transportcited by 6
2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
2025Exploratory Diffusion Model for Unsupervised Reinforcement Learningcited by 5
2025Human-in-the-loop Online Rejection Sampling for Robotic Manipulationcited by 5
2026VGAS: Value-Guided Action-Chunk Selection for Few-Shot Vision-Language-Action Adaptationcited by 5
2026Reinforcement Learning via Value Gradient Flowcited by 5
2026Improving Robotic Generalist Policies via Flow Reversal Steeringcited by 5
2026Beyond Imitation: Self-Improving Robot Policies via Off-Policy Q-Planningcited by 5
2025SpeedAug: Policy Acceleration via Tempo-Enriched Policy and RL Fine-Tuningcited by 4
2026What Makes Value Learning Efficient in Residual Reinforcement Learning?cited by 4
2026Escaping the Diversity Trap in Robotic Manipulation via Anchor-Centric Adaptationcited by 4
2026Look Before You Leap: Distilling Tree Search into Action Evaluation for Frozen VLA Modelscited by 4
2026SOLE-R1: Video-Language Reasoning as the Sole Reward for On-Robot Reinforcement Learningcited by 3
2026Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulationcited by 3
2026Q-VGM: Q-Guided Value-Gradient Matching for Offline-to-Online RL of Flow-Matching VLA Policiescited by 3
2026QPILOTS: Efficient Test-Time Q-Steering for Flow Policiescited by 3
2026Adapting Generalist Robot Policies with Semantic Reinforcement Learningcited by 3
2025Breaking the Performance Ceiling in Reinforcement Learning requires Inference Strategiescited by 2
2025Beyond Success: Refining Elegant Robot Manipulation from Mixed-Quality Data via Just-in-Time Interventioncited by 2
2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
2026Reversal Q-Learningcited by 2
2026FORCE: Efficient VLA Reinforcement Fine-Tuning via Value-Calibrated Warm-up and Self-Distillationcited by 2

50 most cited of 79 All 79 (JSON)

Similar, not cited

Close in wording, no citation either way