# ThinkBot: Embodied Instruction Following with Thought Chain Reasoning

arXiv: https://arxiv.org/abs/2312.07062 · HTML: https://mlsanity.com/paper/2312.07062/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 43 of its 52 references that are in the map.
- First posted on arXiv 12 Dec 2023.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)

## Builds on (most cited first, 30 of 43)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [Inner Monologue: Embodied Reasoning through Planning with Language Models](https://mlsanity.com/paper/2207.05608/) (2022)
- [Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents](https://mlsanity.com/paper/2201.07207/) (2022)
- [AI2-THOR: An Interactive 3D Environment for Visual AI](https://mlsanity.com/paper/1712.05474/) (2017)
- [Large Language Models as Optimizers](https://mlsanity.com/paper/2309.03409/) (2023)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models](https://mlsanity.com/paper/2212.04088/) (2022)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [Ghost in the Minecraft: Generally Capable Agents for Open-World Environments via Large Language Models with Text-based Knowledge and Memory](https://mlsanity.com/paper/2305.17144/) (2023)
- [NavGPT: Explicit Reasoning in Vision-and-Language Navigation with Large Language Models](https://mlsanity.com/paper/2305.16986/) (2023)
- [EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thought](https://mlsanity.com/paper/2305.15021/) (2023)
- [Episodic Transformer for Vision-and-Language Navigation](https://mlsanity.com/paper/2105.06453/) (2021)
- [IQA: Visual Question Answering in Interactive Environments](https://mlsanity.com/paper/1712.03316/) (2017)
- [ESC: Exploration with Soft Commonsense Constraints for Zero-shot Object Navigation](https://mlsanity.com/paper/2301.13166/) (2023)
- [FILM: Following Instructions in Language with Modular Methods](https://mlsanity.com/paper/2110.07342/) (2021)
- [L3MVN: Leveraging Large Language Models for Visual Target Navigation](https://mlsanity.com/paper/2304.05501/) (2023)
- [Discuss Before Moving: Visual Language Navigation via Multi-expert Discussions](https://mlsanity.com/paper/2309.11382/) (2023)
- [A Persistent Spatial Semantic Representation for High-level Natural Language Instruction Execution](https://mlsanity.com/paper/2107.05612/) (2021)
- [Cross-modal Map Learning for Vision and Language Navigation](https://mlsanity.com/paper/2203.05137/) (2022)
- [Weakly-Supervised Multi-Granularity Map Learning for Vision-and-Language Navigation](https://mlsanity.com/paper/2210.07506/) (2022)

## Built on by (most cited first, 19 of 19)

- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [A Survey on Vision-Language-Action Models for Embodied AI](https://mlsanity.com/paper/2405.14093/) (2024)
- [VLA-RL: Towards Masterful and General Robotic Manipulation with Scalable Reinforcement Learning](https://mlsanity.com/paper/2505.18719/) (2025)
- [ManiGaussian: Dynamic Gaussian Splatting for Multi-task Robotic Manipulation](https://mlsanity.com/paper/2403.08321/) (2024)
- [UniVG-R1: Reasoning Guided Universal Visual Grounding with Reinforcement Learning](https://mlsanity.com/paper/2505.14231/) (2025)
- [Vision-and-Language Navigation Today and Tomorrow: A Survey in the Era of Foundation Models](https://mlsanity.com/paper/2407.07035/) (2024)
- [World Modeling Makes a Better Planner: Dual Preference Optimization for Embodied Task Planning](https://mlsanity.com/paper/2503.10480/) (2025)
- [S-Agents: Self-organizing Agents in Open-ended Environments](https://mlsanity.com/paper/2402.04578/) (2024)
- [Ponder & Press: Advancing Visual GUI Agent towards General Computer Control](https://mlsanity.com/paper/2412.01268/) (2024)
- [GAF: Gaussian Action Field as a 4D Representation for Dynamic World Modeling in Robotic Manipulation](https://mlsanity.com/paper/2506.14135/) (2025)
- [Embodied Instruction Following in Unknown Environments](https://mlsanity.com/paper/2406.11818/) (2024)
- [Socratic Planner: Self-QA-Based Zero-Shot Planning for Embodied Instruction Following](https://mlsanity.com/paper/2404.15190/) (2024)
- [ReFineVLA: Reasoning-Aware Teacher-Guided Transfer Fine-Tuning](https://mlsanity.com/paper/2505.19080/) (2025)
- [Stepping Out of Similar Semantic Space for Open-Vocabulary Segmentation](https://mlsanity.com/paper/2506.16058/) (2025)
- [Astra: Efficient Transformer Architecture and Contrastive Dynamics Learning for Embodied Instruction Following](https://mlsanity.com/paper/2408.01147/) (2024)
- [Think Twice, Act Once: Verifier-Guided Action Selection For Embodied Agents](https://mlsanity.com/paper/2605.12620/) (2026)
- [ESCAPE: Episodic Spatial Memory and Adaptive Execution Policy for Long-Horizon Mobile Manipulation](https://mlsanity.com/paper/2604.13633/) (2026)
- [Mini-BEHAVIOR-Gran: Revealing U-Shaped Effects of Instruction Granularity on Language-Guided Embodied Agents](https://mlsanity.com/paper/2604.17019/) (2026)
- [ReFineVLA: Multimodal Reasoning-Aware Generalist Robotic Policies via Teacher-Guided Fine-Tuning](https://mlsanity.com/paper/2604.17800/) (2026)

## Abstract (arXiv)

Embodied Instruction Following (EIF) requires agents to complete human instruction by interacting objects in complicated surrounding environments. Conventional methods directly consider the sparse human instruction to generate action plans for agents, which usually fail to achieve human goals because of the instruction incoherence in action descriptions. On the contrary, we propose ThinkBot that reasons the thought chain in human instruction to recover the missing action descriptions, so that the agent can successfully complete human goals by following the coherent instruction. Specifically, we first design an instruction completer based on large language models to recover the missing actions with interacted objects between consecutive human instruction, where the perceived surrounding environments and the completed sub-goals are considered for instruction completion. Based on the partially observed scene semantic maps, we present an object localizer to infer the position of interacted objects for agents to achieve complex human goals. Extensive experiments in the simulated environment show that our ThinkBot outperforms the state-of-the-art EIF methods by a sizable margin in both success rate and execution efficiency.