mlsanityData to 7 Oct 2026

Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learning

arXiv 2409.06985 ↗First posted 11 Sep 2024cs.LGcited by 3

On the map: Offline RL and imitation learning from data → Decision transformers and sequence modeling

Abstract (arXiv)

Recently, incorporating knowledge from pretrained language models (PLMs) into decision transformers (DTs) has generated significant attention in offline reinforcement learning (RL). These PLMs perform well in RL tasks, raising an intriguing question: what kind of knowledge from PLMs has been transferred to RL to achieve such good results? This work first dives into this problem by analyzing each head quantitatively and points out Markov head, a crucial component that exists in the attention heads of PLMs. It leads to extreme attention on the last-input token and performs well only in short-term environments. Furthermore, we prove that this extreme attention cannot be changed by re-training embedding layer or fine-tuning. Inspired by our analysis, we propose a general method GPT2-DTMA, which equips a pretrained DT with Mixture of Attention (MoA), to accommodate diverse attention requirements during fine-tuning. Extensive experiments corroborate our theorems and demonstrate the effectiveness of GPT2-DTMA: it achieves comparable performance in short-term environments while significantly narrowing the performance gap in long-term environments.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Decision transformers and sequence modeling, one of 13 directions in Offline RL and imitation learning from data. Builds on 32 of its 39 references that are in the map. First posted on arXiv 11 Sep 2024.

Builds on

32 of 39 references.

Closest ancestors · 15

Same direction, newest first

  1. 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer2 Aug
  2. 2023Decision ConvFormer: Local Filtering in MetaFormer is Sufficient for Decision Making4 Oct
  3. 2023Waypoint Transformer: Reinforcement Learning via Supervised Learning with Intermediate Targets24 Jun
  4. 2023Future-conditioned Unsupervised Pretraining for Decision Transformer26 May
  5. 2023Masked Trajectory Models for Prediction, Representation, and Control4 May
  6. 2022Masked Autoencoding for Scalable and Generalizable Decision Making23 Nov
  7. 2022On the Effect of Pre-training for Transformer in Different Modality on Offline Reinforcement Learning17 Nov
  8. 2022Hierarchical Decision Transformer21 Sep
  9. 2022Q-learning Decision Transformer: Leveraging Dynamic Programming for Conditional Sequence Modelling in Offline RL8 Sep
  10. 2022Prompting Decision Transformer for Few-Shot Policy Generalization27 Jun
  11. 2022Online Decision Transformer11 Feb
  12. 2022Can Wikipedia Help Offline Reinforcement Learning?28 Jan
3 more
  1. 2021Generalized Decision Transformer for Offline Hindsight Information Matching19 Nov
  2. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problem3 Jun
  3. 2021Decision Transformer: Reinforcement Learning via Sequence Modeling2 Jun

Other references · 17

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  5. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  6. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  7. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  8. 2019Are Sixteen Heads Really Better than One?cited by 803
  9. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  10. 2024Large Language Models Can Learn Temporal Reasoningcited by 110
  11. 2022Mixture of Attention Heads: Selecting Attention Heads Per Tokencited by 58
  12. 2021StARformer: Transformer with State-Action-Reward Representations for Visual Reinforcement Learningcited by 27
5 more
  1. 2022UniMASK: Unified Inference in Sequential Decision Problemscited by 23
  2. 2021Pretrained Language Models are Symbolic Mathematics Solvers too!cited by 17
  3. 2022Switch Trajectory Transformer with Distributional Value Approximation for Multi-Task Reinforcement Learningcited by 11
  4. 2023Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actionscited by 8
  5. 2022Pre-Trained Language Transformers are Universal Image Classifierscited by 1

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 2.

3 papers.

  1. 2025Anomaly Detection and Early Warning Mechanism for Intelligent Monitoring Systems in Multi-Cloud Environments Based on LLMcited by 1
  2. 2024ResidualDroppath: Enhancing Feature Reuse over Residual Connectionscited by 0
  3. 2025In-Context Compositional Q-Learning for Offline Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Unleashing the Power of Pre-trained Language Models for Offline Reinforcement Learning
  2. 2022Contextual Transformer for Offline Meta Reinforcement Learning
  3. 2026Breaking the Capability Ceiling of LLM Post-Training by Reintroducing Markov States
  4. 2024MORE-3S:Multimodal-based Offline Reinforcement Learning with Shared Semantic Spaces
  5. 2021Representation Matters: Offline Pretraining for Sequential Decision Making
  6. 2025Learning to Focus: Prioritizing Informative Histories with Structured Attention Mechanisms in Partially Observable Reinforcement Learning
  7. 2026QHyer: Q-conditioned Hybrid Attention-mamba Transformer for Offline Goal-conditioned RL
  8. 2025ImagineBench: Evaluating Reinforcement Learning with Large Language Model Rollouts