mlsanityData to 7 Oct 2026

Offline Actor-Critic Reinforcement Learning Scales to Large Models

arXiv 2402.05546 ↗First posted 8 Feb 2024cs.LG cs.AI cs.ROcited by 30

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

We show that offline actor-critic reinforcement learning can scale to large models - such as transformers - and follows similar scaling laws as supervised learning. We find that offline actor-critic algorithms can outperform strong, supervised, behavioral cloning baselines for multi-task training on a large dataset containing both sub-optimal and expert behavior on 132 continuous control tasks. We introduce a Perceiver-based actor-critic model and elucidate the key model features needed to make offline RL work with self- and cross-attention modules. Overall, we find that: i) simple offline actor critic algorithms are a natural choice for gradually moving away from the currently predominant paradigm of behavioral cloning, and ii) via offline RL it is possible to learn multi-task policies that master many domains simultaneously, including real robotics tasks, from sub-optimal demonstrations or self-generated data.

Key facts. Cited by 30 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 45 of its 50 references that are in the map. First posted on arXiv 8 Feb 2024.

Builds on

45 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2022Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes28 Nov
  2. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  3. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning15 Jun
  4. 2021A Minimalist Approach to Offline Reinforcement Learning12 Jun
  5. 2020Critic Regularized Regression26 Jun
  6. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  7. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May
  8. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  8. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  9. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  10. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  11. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  12. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
25 more
  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  3. 2023Open X-Embodiment: Robotic Learning Datasets and RT-X Modelscited by 849
  4. 2022A Generalist Agentcited by 802
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  8. 2021Perceiver IO: A General Architecture for Structured Inputs & Outputscited by 616
  9. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  10. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  11. 2023Scaling up GANs for Text-to-Image Synthesiscited by 492
  12. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  13. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  14. 2018Maximum a Posteriori Policy Optimisationcited by 318
  15. 2022Multi-Game Decision Transformerscited by 212
  16. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 164
  17. 2023Vision-Language Models as Success Detectorscited by 138
  18. 2023Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functionscited by 129
  19. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  20. 2023GNFactor: Multi-Task Real Robot Learning with Generalizable Neural Feature Fieldscited by 99
  21. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  22. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  23. 2023Scaling laws for single-agent reinforcement learningcited by 39
  24. 2022Scaling Laws for a Multi-Agent Reinforcement Learning Modelcited by 37
  25. 2023Mastering Stacking of Diverse Shapes with Large-Scale Iterative Reinforcement Learning on Real Robotscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 8; 2025: 19; 2026: 3.

30 papers.

Sort:

By area:

  1. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  2. 2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
  3. 2025Horizon Reduction Makes RL Scalablecited by 55
  4. 2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
  5. 2025Video2Policy: Scaling up Manipulation Tasks in Simulation through Internet Videoscited by 21
  6. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  7. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  8. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  9. 2025Value-Based Deep RL Scales Predictablycited by 14
  10. 2025Retrospex: Language Agent Meets Offline Reinforcement Learning Criticcited by 12
  11. 2025Scalable Offline Model-Based RL with Action Chunkscited by 12
  12. 2024Coarse-to-fine Q-Network with Action Sequence for Data-Efficient Reinforcement Learningcited by 10
18 more
  1. 2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 9
  2. 2024DemoStart: Demonstration-led auto-curriculum applied to sim-to-real with multi-fingered robotscited by 7
  3. 2024AlphaZero Neural Scaling and Zipf's Law: a Tale of Board Games and Power Lawscited by 7
  4. 2025Compute-Optimal Scaling for Value-Based Deep RLcited by 5
  5. 2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
  6. 2025Human-Level Competitive Pok'emon via Scalable Offline Reinforcement Learning with Transformerscited by 4
  7. 2025So-Fake: Benchmarking and Explaining Social Media Image Forgery Detectioncited by 4
  8. 2025What does really matter in image goal navigation?cited by 2
  9. 2025Kinaema: a recurrent sequence model for memory and pose in motioncited by 2
  10. 2025Exploiting Policy Idling for Dexterous Manipulationcited by 1
  11. 2025Information-Theoretic Policy Pre-Training with Empowermentcited by 1
  12. 2024CLSP: High-Fidelity Contrastive Language-State Pre-training for Agent State Representationcited by 0
  13. 2025A recurrent vision transformer shows signatures of primate visual attentioncited by 0
  14. 2025Value from Observations: Towards Large-Scale Imitation Learning via Self-Improvementcited by 0
  15. 2025Scaling DRL for Decision Making: A Survey on Data, Network, and Training Budget Strategiescited by 0
  16. 2026Cross-Embodiment Offline Reinforcement Learning for Heterogeneous Robot Datasetscited by 0
  17. 2026Horizon Adaptive Offline Policy Learning via Value Stitchingcited by 0
  18. 2026SeeQ: Training Generalist Value Functions for Long-Horizon Robotic Manipulationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learning
  2. 2024Efficient Offline Reinforcement Learning: First Imitate, then Improve
  3. 2026Decoupling Policy Extraction for Offline Reinforcement Learning
  4. 2021Provable Benefits of Actor-Critic Methods for Offline Reinforcement Learning
  5. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learning
  6. 2026Proximal Action Replacement for Behavior Cloning Actor-Critic in Offline Reinforcement Learning
  7. 2026Flow Actor-Critic for Offline Reinforcement Learning
  8. 2023Importance Weighted Actor-Critic for Optimal Conservative Offline Reinforcement Learning