mlsanityData to 7 Oct 2026

Q-Transformer: Scalable Offline Reinforcement Learning via Autoregressive Q-Functions

arXiv 2309.10150 ↗First posted 18 Sep 2023cs.RO cs.AI cs.LGcited by 129

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

In this work, we present a scalable reinforcement learning method for training multi-task policies from large offline datasets that can leverage both human demonstrations and autonomously collected data. Our method uses a Transformer to provide a scalable representation for Q-functions trained via offline temporal difference backups. We therefore refer to the method as Q-Transformer. By discretizing each action dimension and representing the Q-value of each action dimension as separate tokens, we can apply effective high-capacity sequence modeling techniques for Q-learning. We present several design decisions that enable good performance with offline RL training, and show that Q-Transformer outperforms prior offline RL algorithms and imitation learning techniques on a large diverse real-world robotic manipulation task suite. The project's website and videos can be found at https://qtransformer.github.io

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 61 of its 78 references that are in the map. First posted on arXiv 18 Sep 2023.

Builds on

61 of 78 references.

Closest ancestors · 4

Same direction, newest first

  1. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  2. 2022Multi-Game Decision Transformers30 May
  3. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  4. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  3. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  4. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  5. 2017FiLM: Visual Reasoning with a General Conditioning Layercited by 2,286
  6. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  7. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  9. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  11. 2017Hindsight Experience Replaycited by 1,096
  12. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,013
38 more
  1. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  2. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  3. 2022A Generalist Agentcited by 802
  4. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  5. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  6. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  7. 2021What Matters in Learning from Offline Human Demonstrations for Robot Manipulationcited by 699
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  10. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  11. 2018On Evaluation of Embodied Navigation Agentscited by 526
  12. 2022BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learningcited by 517
  13. 2018Universal Sentence Encodercited by 485
  14. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  15. 2016Target-driven Visual Navigation in Indoor Scenes using Deep Reinforcement Learningcited by 351
  16. 2022Behavior Transformers: Cloning k modes with one stonecited by 299
  17. 2019Way Off-Policy Batch Deep Reinforcement Learning of Implicit Human Preferences in Dialogcited by 283
  18. 2020Critic Regularized Regressioncited by 269
  19. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  20. 2021MT-Opt: Continuous Multi-Task Robotic Reinforcement Learning at Scalecited by 226
  21. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 213
  22. 2016Learning to Navigate in Complex Environmentscited by 192
  23. 2022GNM: A General Navigation Model to Drive Any Robotcited by 161
  24. 2021Episodic Transformer for Vision-and-Language Navigationcited by 155
  25. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  26. 2021Learning Language-Conditioned Robot Behavior from Offline Data and Crowd-Sourced Annotationcited by 117
  27. 2019Scene Memory Transformer for Embodied Agents in Long-Horizon Taskscited by 106
  28. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  29. 2019Reward-Conditioned Policiescited by 101
  30. 2019BAIL: Best-Action Imitation Learning for Batch Deep Reinforcement Learningcited by 90
  31. 2022When Should We Prefer Offline Reinforcement Learning Over Behavioral Cloning?cited by 86
  32. 2022MetaMorph: Learning Universal Controllers with Transformerscited by 78
  33. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  34. 2020Transformers for One-Shot Visual Imitationcited by 77
  35. 2022When does return-conditioned supervised learning work for offline reinforcement learning?cited by 73
  36. 2021Beyond Pick-and-Place: Tackling Robotic Stacking of Diverse Shapescited by 70
  37. 2021A Workflow for Offline Model-Free Robotic Reinforcement Learningcited by 68
  38. 2021Hierarchical Task Learning from Language Instructions with Unified Transformers and Self-Monitoringcited by 53

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 11; 2024: 48; 2025: 37; 2026: 33.

129 papers.

Sort:

By area:

  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 391
  2. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  3. 2023NoMaD: Goal Masked Diffusion Policies for Navigation and Explorationcited by 234
  4. 2024Aligning Cyber Space with Physical World: A Comprehensive Survey on Embodied AIcited by 175
  5. 2024Deep Reinforcement Learning for Robotics: A Survey of Real-World Successescited by 153
  6. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  7. 2023Large Language Models for Robotics: A Surveycited by 132
  8. 2025RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twinscited by 117
  9. 2024Steering Your Generalists: Improving Robotic Foundation Models via Value Guidancecited by 100
  10. 2023Toward General-Purpose Robots via Foundation Models: A Survey and Meta-Analysiscited by 99
  11. 2024Policy Agnostic RL: Offline RL and Online RL Fine-Tuning of Any Class and Backbonecited by 71
  12. 2024RoboTwin: Dual-Arm Robot Benchmark with Generative Digital Twins (early version)cited by 68
38 more
  1. 2024Real-World Robot Applications of Foundation Models: A Reviewcited by 60
  2. 2025Horizon Reduction Makes RL Scalablecited by 55
  3. 2024A Survey on Robotics with Foundation Models: toward Embodied AIcited by 53
  4. 2025ReinboT: Amplifying Robot Visual-Language Manipulation with Reinforcement Learningcited by 44
  5. 2024Towards Generalist Robot Learning from Internet Video: A Surveycited by 40
  6. 2023Action-Quantized Offline Reinforcement Learning for Robotic Skill Learningcited by 34
  7. 2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
  8. 2024Offline Actor-Critic Reinforcement Learning Scales to Large Modelscited by 30
  9. 2024GeRM: A Generalist Robotic Model with Mixture-of-experts for Quadruped Robotcited by 28
  10. 2024TeleMoMa: A Modular and Versatile Teleoperation System for Mobile Manipulationcited by 27
  11. 2025MoRE: Unlocking Scalability in Reinforcement Learning for Quadruped Vision-Language-Action Modelscited by 27
  12. 2023Towards Robust Offline Reinforcement Learning under Diverse Data Corruptioncited by 24
  13. 2025SRT-H: A Hierarchical Framework for Autonomous Surgery via Language Conditioned Imitation Learningcited by 23
  14. 2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
  15. 2024Jack of All Trades, Master of Some, a Multi-Purpose Transformer Agentcited by 20
  16. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  17. 2025Digi-Q: Learning Q-Value Functions for Training Device-Control Agentscited by 20
  18. 2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
  19. 2024SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulationcited by 18
  20. 2024TOP-ERL: Transformer-based Off-Policy Episodic Reinforcement Learningcited by 18
  21. 2023Reinforcement Learning with Foundation Priors: Let the Embodied Agent Efficiently Learn on Its Owncited by 16
  22. 2024Reinforcing Language Agents via Policy Optimization with Action Decompositioncited by 16
  23. 2025RobustVLA: Robustness-Aware Reinforcement Post-Training for Vision-Language-Action Modelscited by 16
  24. 2024General-purpose foundation models for increased autonomy in robot-assisted surgerycited by 15
  25. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  26. 2024Continuous Control with Coarse-to-fine Reinforcement Learningcited by 15
  27. 2023Rethinking Decision Transformer via Hierarchical Reinforcement Learningcited by 14
  28. 2023LiFT: Unsupervised Reinforcement Learning with Foundation Models as Teacherscited by 14
  29. 2024AMAGO-2: Breaking the Multi-Task Barrier in Meta-Reinforcement Learning with Transformerscited by 14
  30. 2025Temporal Representation Alignment: Successor Features Enable Emergent Compositionality in Robot Instruction Followingcited by 13
  31. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 13
  32. 2024Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcementcited by 12
  33. 2025Exploring Embodied Multimodal Large Models: Development, Datasets, and Future Directionscited by 12
  34. 2025Soft Policy Optimization: Online Off-Policy RL for Sequence Modelscited by 12
  35. 2025A Survey on Imitation Learning for Contact-Rich Tasks in Roboticscited by 12
  36. 2025Planning without Search: Refining Frontier LLMs with Offline Goal-Conditioned RLcited by 11
  37. 2024A2PO: Towards Effective Offline Reinforcement Learning from an Advantage-aware Perspectivecited by 10
  38. 2024IGC-Net for conditional average potential outcome estimation over timecited by 10

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026When Life Gives You BC, Make Q-functions: Extracting Q-values from Behavior Cloning for On-Robot Reinforcement Learning
  2. 2026Q-learning Penalized Transformer for Safe Offline Reinforcement Learning
  3. 2024Q-value Regularized Transformer for Offline Reinforcement Learning
  4. 2026From Weak Data to Strong Policy: Q-Targets Enable Provable In-Context Reinforcement Learning
  5. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learning
  6. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?
  7. 2026Reversal Q-Learning
  8. 2026RankQ: Offline-to-Online Reinforcement Learning via Self-Supervised Action Ranking