Think Before You Act: Unified Policy for Interleaving Language Reasoning with Actions
On the map: LLM reasoning → Reinforcement learning for reasoning
Abstract (arXiv)
The success of transformer models trained with a language modeling objective brings a promising opportunity to the reinforcement learning framework. Decision Transformer is a step towards this direction, showing how to train transformers with a similar next-step prediction objective on offline data. Another important development in this area is the recent emergence of large-scale datasets collected from the internet, such as the ones composed of tutorial videos with captions where people talk about what they are doing. To take advantage of this language component, we propose a novel method for unifying language reasoning with actions in a single policy. Specifically, we augment a transformer policy with word outputs, so it can generate textual captions interleaved with actions. When tested on the most challenging task in BabyAI, with captions describing next subgoals, our reasoning policy consistently outperforms the caption-free baseline.
Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 11 of its 28 references that are in the map. First posted on arXiv 18 Apr 2023.
Builds on
11 of 28 references.
References · 11
- 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
- 2017Decoupled Weight Decay Regularizationcited by 1,831
- 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
- 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 345
- 2018BabyAI: A Platform to Study the Sample Efficiency of Grounded Language Learningcited by 244
- 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221
- 2023Grounding Large Language Models in Interactive Environments with Online Reinforcement Learningcited by 195
- 2020Human Instruction-Following with Deep Reinforcement Learning via Transfer-Learning from Textcited by 59
- 2023A Survey on Transformers in Reinforcement Learningcited by 50
- 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
- 2022Hierarchical Decision Transformercited by 16
Built on by
Citing papers per year: 2023: 1; 2024: 5; 2025: 1; 2026: 1.
8 papers.
- 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 116
- 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 108
- 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinkingcited by 30
- 2024How Can LLM Guide RL? A Value-Based Approachcited by 13
- 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
- 2024Are Visual-Language Models Effective in Action Recognition? A Comparative Studycited by 4
- 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learningcited by 3
- 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidancecited by 0
- 2026MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance20 Feb 2026
- 2025Multi-Agent Collaboration via Evolving Orchestration26 May 2025
- 2024Are Visual-Language Models Effective in Action Recognition? A Comparative Study22 Oct 2024
- 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Review21 Oct 2024
- 2024Unveiling Markov Heads in Pretrained Language Models for Offline Reinforcement Learning11 Sep 2024
- 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methods30 Mar 2024
- 2024How Can LLM Guide RL? A Value-Based Approach25 Feb 2024
- 2023Thought Cloning: Learning to Think while Acting by Imitating Human Thinking1 Jun 2023
Similar, not cited
Close in wording, no citation either way
- 2026Transformer-Based Language Models Across Domain Verticals: Architectures, Applications and Critical Assessment
- 2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility
- 2025Enhancing Decision-Making of Large Language Models via Actor-Critic
- 2026In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use
- 2025Do What You Say: Steering Vision-Language-Action Models via Runtime Reasoning-Action Alignment Verification
- 2024Pre-trained Language Models Improve the Few-shot Prompt Ability of Decision Transformer
- 2021Decision Transformer: Reinforcement Learning via Sequence Modeling
- 2025Natural Language Actor-Critic: Scalable Off-Policy Learning in Language Space