mlsanityData to 7 Oct 2026

Can Wikipedia Help Offline Reinforcement Learning?

arXiv 2201.12122 ↗First posted 28 Jan 2022cs.LG cs.AI cs.CLcited by 76

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Fine-tuning reinforcement learning (RL) models has been challenging because of a lack of large scale off-the-shelf datasets as well as high variance in transferability among different environments. Recent work has looked at tackling offline RL from the perspective of sequence modeling with improved results as result of the introduction of the Transformer architecture. However, when the model is trained from scratch, it suffers from slow convergence speeds. In this paper, we look to take advantage of this formulation of reinforcement learning as sequence modeling and investigate the transferability of pre-trained sequence models on other domains (vision, language) when finetuned on offline RL tasks (control, games). To this end, we also propose techniques to improve transfer between these domains. Results show consistent performance gains in terms of both convergence speed and reward on a variety of environments, accelerating training by 3-6x and achieving state-of-the-art performance in a variety of tasks using Wikipedia-pretrained and GPT2 language models. We hope that this work not only brings light to the potentials of leveraging generic sequence modeling techniques and pre-trained models for RL, but also inspires future work on sharing knowledge between generative modeling tasks of completely different domains.

Key facts. Cited by 76 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 53 of its 72 references that are in the map. First posted on arXiv 28 Jan 2022.

Builds on

53 of 72 references.

Closest ancestors · 2

Same direction, newest first

  1. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  2. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2016Layer Normalizationcited by 5,579
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2015Continuous control with deep reinforcement learningcited by 3,279
  11. 2016Pointer Sentinel Mixture Modelscited by 2,992
  12. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
38 more
  1. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  6. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  7. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  8. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  9. 2020Mastering Atari with Discrete World Modelscited by 990
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  12. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  13. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  14. 2018QT-Opt: Scalable Deep Reinforcement Learning for Vision-Based Robotic Manipulationcited by 730
  15. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  16. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  17. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 617
  18. 2020robosuite: A Modular Simulation Framework and Benchmark for Robot Learningcited by 484
  19. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 478
  20. 2016Deep Reinforcement Learning for Robotic Manipulation with Asynchronous Off-Policy Updatescited by 342
  21. 2020Transporter Networks: Rearranging the Visual World for Robotic Manipulationcited by 342
  22. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  23. 2018Adaptive Input Representations for Neural Language Modelingcited by 226
  24. 2020Language Conditioned Imitation Learning over Unstructured Datacited by 208
  25. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  26. 2016Continuous Deep Q-Learning with Model-based Accelerationcited by 149
  27. 2019Language as an Abstraction for Hierarchical Deep Reinforcement Learningcited by 138
  28. 2020Parrot: Data-Driven Behavioral Priors for Reinforcement Learningcited by 136
  29. 2021Actionable Models: Unsupervised Offline Reinforcement Learning of Robotic Skillscited by 125
  30. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 113
  31. 2019Training Agents using Upside-Down Reinforcement Learningcited by 106
  32. 2019Reward-Conditioned Policiescited by 101
  33. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  34. 2019Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularitycited by 68
  35. 2020My Body is a Cage: the Role of Morphology in Graph-Based Incompatible Controlcited by 68
  36. 2021Learning to See before Learning to Act: Visual Pre-training for Manipulationcited by 50
  37. 2018Hardware Conditioned Policies for Multi-Robot Transfer Learningcited by 42
  38. 2020Behavior Priors for Efficient Reinforcement Learningcited by 40

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 31; 2023: 28; 2024: 14; 2025: 1; 2026: 2.

76 papers.

Sort:

By area:

  1. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
  2. 2022A Generalist Agentcited by 802
  3. 2022MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledgecited by 469
  4. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 403
  5. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  6. 2022Data Distributional Properties Drive Emergent In-Context Learning in Transformerscited by 303
  7. 2023Generating Images with Multimodal Language Modelscited by 259
  8. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  9. 2022Multi-Game Decision Transformerscited by 212
  10. 2023Large Language Models as General Pattern Machinescited by 196
  11. 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 193
  12. 2023Language-Driven Representation Learning for Roboticscited by 163
38 more
  1. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  2. 2023Human-Timescale Adaptation in an Open-Ended Task Spacecited by 136
  3. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  4. 2023Mechanistically analyzing the effects of fine-tuning on procedurally defined taskscited by 110
  5. 2023Grounded Decoding: Guiding Text Generation with Grounded Models for Embodied Agentscited by 66
  6. 2022You Can't Count on Luck: Why Decision Transformers and RvS Fail in Stochastic Environmentscited by 62
  7. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  8. 2022Semantic Exploration from Language Abstractions and Pretrained Representationscited by 51
  9. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  10. 2022Dichotomy of Control: Separating What You Can Control from What You Cannotcited by 47
  11. 2023Can Language Models Learn to Listen?cited by 40
  12. 2022History Compression via Language Models in Reinforcement Learningcited by 39
  13. 2023Generalization to New Sequential Decision Making Tasks with In-Context Learningcited by 39
  14. 2022Bootstrapped Transformer for Offline Reinforcement Learningcited by 37
  15. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35
  16. 2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
  17. 2022Disentangling Transfer in Continual Reinforcement Learningcited by 31
  18. 2023Interactive Natural Language Processingcited by 31
  19. 2022Chain of Thought Imitation with Procedure Cloningcited by 30
  20. 2022Pretraining with Artificial Language: Studying Transferable Knowledge in Language Modelscited by 28
  21. 2022MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learnedcited by 26
  22. 2023Decision S4: Efficient Sequence-Based RL via State Spaces Layerscited by 26
  23. 2022Insights into Pre-training via Simpler Synthetic Taskscited by 25
  24. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learningcited by 24
  25. 2022Law Informs Code: A Legal Informatics Approach to Aligning Artificial Intelligence with Humanscited by 23
  26. 2022Generating Executable Action Plans with Environmentally-Aware Language Modelscited by 21
  27. 2022Downstream Datasets Make Surprisingly Good Pretraining Corporacited by 20
  28. 2022Q-Ensemble for Offline RL: Don't Scale the Ensemble, Scale the Batch Sizecited by 20
  29. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  30. 2024DecisionNCE: Embodied Multimodal Representations via Implicit Preference Learningcited by 19
  31. 2022Pretraining in Deep Reinforcement Learning: A Surveycited by 18
  32. 2023Semantic HELM: A Human-Readable Memory for Reinforcement Learningcited by 17
  33. 2024From Words to Actions: Unveiling the Theoretical Underpinnings of LLM-Driven Autonomous Systemscited by 17
  34. 2022Hierarchical Decision Transformercited by 16
  35. 2024Efficient Sequential Decision Making with Large Language Modelscited by 16
  36. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  37. 2024The RL/LLM Taxonomy Tree: Reviewing Synergies Between Reinforcement Learning and Large Language Modelscited by 15
  38. 2026Training Language Models via Neural Cellular Automatacited by 15

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem
  3. 2023Model-Based Reinforcement Learning with Multi-Task Offline Pretraining
  4. 2023Finetuning from Offline Reinforcement Learning: Challenges, Trade-offs and Practical Solutions
  5. 2023The Generalization Gap in Offline Reinforcement Learning
  6. 2023Finetuning Offline World Models in the Real World
  7. 2025A Clean Slate for Offline Reinforcement Learning
  8. 2025Efficient Reinforcement Learning by Guiding World Models with Non-Curated Data