mlsanityData to 7 Oct 2026

Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning

arXiv 2310.20587 ↗First posted 31 Oct 2023cs.LGcited by 24

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Offline reinforcement learning (RL) aims to find a near-optimal policy using pre-collected datasets. In real-world scenarios, data collection could be costly and risky; therefore, offline RL becomes particularly challenging when the in-domain data is limited. Given recent advances in Large Language Models (LLMs) and their few-shot learning prowess, this paper introduces Language Models for Motion Control (LaMo), a general framework based on Decision Transformers to effectively use pre-trained Language Models (LMs) for offline RL. Our framework highlights four crucial components: (1) Initializing Decision Transformers with sequentially pre-trained LMs, (2) employing the LoRA fine-tuning method, in contrast to full-weight fine-tuning, to combine the pre-trained knowledge from LMs and in-domain knowledge effectively, (3) using the non-linear MLP transformation instead of linear projections, to generate embeddings, and (4) integrating an auxiliary language prediction loss during fine-tuning to stabilize the LMs and retain their original abilities on languages. Empirical results indicate LaMo achieves excellent performance in sparse-reward tasks and closes the gap between value-based offline RL methods and decision transformers in dense-reward tasks. In particular, our method demonstrates superior performance in scenarios with limited data samples.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 59 of its 69 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

59 of 69 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  2. 2023When should we prefer Decision Transformers for Offline Reinforcement Learning?23 May
  3. 2023Decision Transformer under Random Frame Dropping3 Mar
  4. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  5. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  6. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
  7. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 51

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2022Training language models to follow instructions with human feedbackcited by 15,712
  6. 2017Proximal Policy Optimization Algorithmscited by 14,441
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  10. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  11. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
38 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  4. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  5. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2022RT-1: Robotics Transformer for Real-World Control at Scalecited by 2,007
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  13. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  14. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,029
  15. 2021Transformers in Vision: A Surveycited by 1,009
  16. 2020Mastering Atari with Discrete World Modelscited by 990
  17. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  18. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  19. 2022A Generalist Agentcited by 802
  20. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  21. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  22. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learningcited by 543
  23. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  24. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  25. 2023Text2Motion: From Natural Language Instructions to Feasible Planscited by 223
  26. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 220
  27. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  28. 2022In-context Reinforcement Learning with Algorithm Distillationcited by 186
  29. 2023Revisiting the Minimalist Approach to Offline Reinforcement Learningcited by 121
  30. 2022CORL: Research-oriented Deep Offline Reinforcement Learning Librarycited by 117
  31. 2021d3rlpy: An Offline Deep Reinforcement Learning Librarycited by 84
  32. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  33. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  34. 2023Prompt a Robot to Walk with Large Language Modelscited by 44
  35. 2023SayTap: Language to Quadrupedal Locomotioncited by 40
  36. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  37. 2023Prompt-Tuning Decision Transformer with Preference Rankingcited by 19
  38. 2020Reservoir Transformerscited by 18

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 7; 2026: 2.

24 papers.

Sort:

By area:

  1. 2024TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulationcited by 331
  2. 2024Agent AI: Surveying the Horizons of Multimodal Interactioncited by 167
  3. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  4. 2024From Multimodal LLMs to Generalist Embodied Agents: Methods and Lessonscited by 42
  5. 2024Grounding Multimodal Large Language Models in Actionscited by 35
  6. 2024iLLM-TSC: Integration reinforcement learning and large language model for traffic signal control policy improvementcited by 23
  7. 2025Parallels Between VLA Model Post-Training and Human Motor Learning: Progress, Challenges, and Trendscited by 17
  8. 2024Decision Mamba: A Multi-Grained State Space Model with Self-Evolution Regularization for Offline RLcited by 15
  9. 2024Stop Wandering, Find the Keys: LLMs Discriminate Key States for Efficient Multi-Agent Explorationcited by 9
  10. 2024Tackling Data Corruption in Offline Reinforcement Learning via Sequence Modelingcited by 7
  11. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformercited by 6
  12. 2024Zero-shot Model-based Reinforcement Learning using Large Language Modelscited by 6
12 more
  1. 2024Decision Transformer as a Foundation Model for Partially Observable Continuous Controlcited by 5
  2. 2025Text-to-Decision Agent: Offline Meta-Reinforcement Learning from Natural Language Supervisioncited by 5
  3. 2025HCRMP: A LLM-Hinted Contextual Reinforcement Learning Framework for Autonomous Drivingcited by 4
  4. 2024Robust RL with LLM-Driven Data Synthesis and Policy Adaptation for Autonomous Drivingcited by 3
  5. 2024Guiding Reinforcement Learning Using Uncertainty-Aware Large Language Modelscited by 3
  6. 2025Prompt Tuning Decision Transformers with Structured and Scalable Banditscited by 2
  7. 2025Large Language Model-enhanced Reinforcement Learning for Low-Altitude Economy Networkingcited by 2
  8. 2025ADG: Ambient Diffusion-Guided Dataset Recovery for Corruption-Robust Offline Reinforcement Learningcited by 2
  9. 2026LLM-Guided Communication for Cooperative Multi-Agent Reinforcement Learningcited by 2
  10. 2025Agentic Episodic Controlcited by 1
  11. 2024Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spacescited by 0
  12. 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learning
  2. 2023Leftover Lunch: Advantage-based Offline Reinforcement Learning for Language Models
  3. 2024MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spaces
  4. 2025Exploring the Potential of Offline RL for Reasoning in LLMs: A Preliminary Study
  5. 2026Leveraging Offline Supervision for Efficient and Generalizable Reinforcement Learning in Large-Scale Vision-Language-Action Models
  6. 2024Q-SFT: Q-Learning for Language Models via Supervised Fine-Tuning
  7. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  8. 2025Recall-Extend Dynamics: Enhancing Small Language Models through Controlled Exploration and Refined Offline Integration