Data to 14 Oct 2026

Offline Q-Learning on Diverse Multi-Task Data Both Scales And Generalizes

arXiv 2211.15144 ↗First posted 28 Nov 2022cs.LGcited by 73

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

The potential of offline reinforcement learning (RL) is that high-capacity models trained on large, heterogeneous datasets can lead to agents that generalize broadly, analogously to similar advances in vision and NLP. However, recent works argue that offline RL methods encounter unique challenges to scaling up model capacity. Drawing on the learnings from these works, we re-examine previous design choices and find that with appropriate choices: ResNets, cross-entropy based distributional backups, and feature normalization, offline Q-learning algorithms exhibit strong performance that scales with model capacity. Using multi-task Atari as a testbed for scaling and generalization, we train a single policy on 40 games with near-human performance using up-to 80 million parameter networks, finding that model performance scales favorably with capacity. In contrast to prior work, we extrapolate beyond dataset performance even when trained entirely on a large (400M transitions) but highly suboptimal dataset (51% human-level performance). Compared to return-conditioned supervised approaches, offline Q-learning scales similarly with model capacity and has better performance, especially when the dataset is suboptimal. Finally, we show that offline Q-learning with a diverse dataset is sufficient to learn powerful representations that facilitate rapid transfer to novel games and fast online learning on new variations of a training game, improving over existing state-of-the-art representation learning approaches.

Key facts. Cited by 73 papers in the map (data through 14 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 54 of its 75 references that are in the map. First posted on arXiv 28 Nov 2022.

Builds on

54 of 75 references.

Closest ancestors · 14

Same direction, newest first

2 more

2019Behavior Regularized Offline Reinforcement Learning26 Nov
2019Benchmarking Batch Deep Reinforcement Learning Algorithms3 Oct

Other references · 40

2017Attention Is All You Needcited by 60,361
2015Deep Residual Learning for Image Recognitioncited by 52,096
2020Language Models are Few-Shot Learnerscited by 32,419
2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
2018Representation Learning with Contrastive Predictive Codingcited by 7,777
2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
2020Scaling Laws for Neural Language Modelscited by 6,586
2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
2013Playing Atari with Deep Reinforcement Learningcited by 3,274
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
2018Group Normalizationcited by 1,524
2020Gradient Surgery for Multi-Task Learningcited by 1,256

28 more

2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
2022A Generalist Agentcited by 808
2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
2017A Distributional Perspective on Reinforcement Learningcited by 765
2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
2015Policy Distillationcited by 440
2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
2022Masked Visual Pre-training for Motor Controlcited by 248
2017Distral: Robust Multitask Reinforcement Learningcited by 233
2022Multi-Game Decision Transformerscited by 214
2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
2015Actor-Mimic: Deep Multitask and Transfer Reinforcement Learningcited by 178
2018Deep Reinforcement Learning and the Deadly Triadcited by 174
2018Multi-task Deep Reinforcement Learning with PopArtcited by 141
2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 122
2018Generalization and Regularization in DQNcited by 118
2019V-MPO: On-Policy Maximum a Posteriori Policy Optimization for Discrete and Continuous Controlcited by 85
2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
2022In Defense of the Unitary Scalarization for Deep Multi-Task Learningcited by 67
2022Reincarnating Reinforcement Learning: Reusing Prior Computation to Accelerate Progresscited by 65
2021SCENIC: A JAX Library for Computer Vision Research and Beyondcited by 57
2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 42
2021Training Larger Networks for Deep Reinforcement Learningcited by 38
2018Visual Transfer between Atari Games using Competitive Reinforcement Learningcited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 19; 2024: 19; 2025: 24; 2026: 11.

73 papers.

Sort:

By area:

2023Learning Universal Policies via Text-Guided Video Generationcited by 522
2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 144
2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 126
2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learningcited by 115
2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous controlcited by 108
2023STEVE-1: A Generative Model for Text-to-Behavior in Minecraftcited by 97
2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 78
2025Horizon Reduction Makes RL Scalablecited by 57
2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 56

38 more

2023Anti-Exploration by Random Network Distillationcited by 50
20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
2024True Knowledge Comes from Practice: Aligning LLMs with Embodied Environments via Reinforcement Learningcited by 36
2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
2023The Generalization Gap in Offline Reinforcement Learningcited by 26
2024Reinformer: Max-Return Sequence Modeling for Offline RLcited by 25
2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 23
2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 23
2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 22
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2023Understanding, Predicting and Better Resolving Q-Value Divergence in Offline-RLcited by 20
2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
2023Searching for High-Value Molecules Using Reinforcement Learning and Transformerscited by 18
2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
2024Contrastive Imitation Learning for Language-guided Multi-Task Robotic Manipulationcited by 15
2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 15
2025Value-Based Deep RL Scales Predictablycited by 15
2026Learning While Deploying: Fleet-Scale Reinforcement Learning for Generalist Robot Policiescited by 15
2024Bayesian Design Principles for Offline-to-Online Reinforcement Learningcited by 14
2024Scaling Offline Model-Based RL via Jointly-Optimized World-Action Model Pretrainingcited by 14
2024Don't flatten, tokenize! Unlocking the key to SoftMoE's efficacy in deep RLcited by 14
2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 13
2025Scalable Offline Model-Based RL with Action Chunkscited by 13
2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
2025Learning Massively Multitask World Models for Continuous Controlcited by 12
2025DEAS: DEtached value learning with Action Sequence for Scalable Offline RLcited by 10
2023Making Offline RL Online: Collaborative World Models for Offline Visual Reinforcement Learningcited by 7
2023Robotic Manipulation Datasets for Offline Compositional Reinforcement Learningcited by 6
2023Hybrid Search for Efficient Planning with Completeness Guaranteescited by 6
2025Compute-Optimal Scaling for Value-Based Deep RLcited by 6
2023Katakomba: Tools and Benchmarks for Data-Driven NetHackcited by 5
2023Budgeting Counterfactual for Offline RLcited by 4
2024Investigating Pre-Training Objectives for Generalization in Vision-Based Reinforcement Learningcited by 4
2024Is Value Functions Estimation with Classification Plug-and-play for Offline Reinforcement Learning?cited by 4
2024Q-value Regularized Decision ConvFormer for Offline Reinforcement Learningcited by 4

50 most cited of 73 All 73 (JSON)

Similar, not cited

Close in wording, no citation either way