# Chain-of-Thought Predictive Control

arXiv: https://arxiv.org/abs/2304.00776 · HTML: https://mlsanity.com/paper/2304.00776/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 83 of its 108 references that are in the map.
- First posted on arXiv 3 Apr 2023.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Offline RL and imitation learning from data / [Robot manipulation imitation learning](https://mlsanity.com/offlinerl/robot-manipulation-imitation-learning/)

## Builds on (most cited first, 30 of 83)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [End-to-End Object Detection with Transformers](https://mlsanity.com/paper/2005.12872/) (2020)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation](https://mlsanity.com/paper/1612.00593/) (2016)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling](https://mlsanity.com/paper/1412.3555/) (2014)
- [Do As I Can, Not As I Say: Grounding Language in Robotic Affordances](https://mlsanity.com/paper/2204.01691/) (2022)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [PaLM-E: An Embodied Multimodal Language Model](https://mlsanity.com/paper/2303.03378/) (2023)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [Planning with Diffusion for Flexible Behavior Synthesis](https://mlsanity.com/paper/2205.09991/) (2022)
- [DeepMind Control Suite](https://mlsanity.com/paper/1801.00690/) (2018)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reduction](https://mlsanity.com/paper/1906.00949/) (2019)
- [ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Tasks](https://mlsanity.com/paper/1912.01734/) (2019)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [What Matters in Learning from Offline Human Demonstrations for Robot Manipulation](https://mlsanity.com/paper/2108.03298/) (2021)
- [Learning Complex Dexterous Manipulation with Deep Reinforcement Learning and Demonstrations](https://mlsanity.com/paper/1709.10087/) (2017)
- [AWAC: Accelerating Online Reinforcement Learning with Offline Datasets](https://mlsanity.com/paper/2006.09359/) (2020)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)

## Built on by (most cited first, 18 of 18)

- [Towards Better Chain-of-Thought Prompting Strategies: A Survey](https://mlsanity.com/paper/2310.04959/) (2023)
- [Point Cloud Matters: Rethinking the Impact of Different Observation Spaces on Robot Learning](https://mlsanity.com/paper/2402.02500/) (2024)
- [Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learning](https://mlsanity.com/paper/2402.03570/) (2024)
- [Play to the Score: Stage-Guided Dynamic Multi-Sensory Fusion for Robotic Manipulation](https://mlsanity.com/paper/2408.01366/) (2024)
- [Kinematic-aware Prompting for Generalizable Articulated Object Manipulation with LLMs](https://mlsanity.com/paper/2311.02847/) (2023)
- [SAM-E: Leveraging Visual Foundation Model with Sequence Imitation for Embodied Manipulation](https://mlsanity.com/paper/2405.19586/) (2024)
- [EasyHeC: Accurate and Automatic Hand-eye Calibration via Differentiable Rendering and Space Exploration](https://mlsanity.com/paper/2305.01191/) (2023)
- [Think Small, Act Big: Primitive Prompt Learning for Lifelong Robot Manipulation](https://mlsanity.com/paper/2504.00420/) (2025)
- [CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding](https://mlsanity.com/paper/2310.06214/) (2023)
- [Efficient Planning with Latent Diffusion](https://mlsanity.com/paper/2310.00311/) (2023)
- [Rethinking Latent Redundancy in Behavior Cloning: An Information Bottleneck Approach for Robot Manipulation](https://mlsanity.com/paper/2502.02853/) (2025)
- [MaxMI: A Maximal Mutual Information Criterion for Manipulation Concept Discovery](https://mlsanity.com/paper/2407.15086/) (2024)
- [InfoCon: Concept Discovery with Generative and Discriminative Informativeness](https://mlsanity.com/paper/2404.10606/) (2024)
- [Multi-State-Action Tokenisation in Decision Transformers for Multi-Discrete Action Spaces](https://mlsanity.com/paper/2407.01310/) (2024)
- [Hierarchical Diffusion Policy: manipulation trajectory generation via contact guidance](https://mlsanity.com/paper/2411.12982/) (2024)
- [M3PC: Test-time Model Predictive Control for Pretrained Masked Trajectory Model](https://mlsanity.com/paper/2412.05675/) (2024)
- [Inverse Reinforcement Learning via Convex Optimization](https://mlsanity.com/paper/2501.15957/) (2025)
- [Reinforcement Learning: From Algorithms To Foundation Models](https://mlsanity.com/paper/2607.17560/) (2026)

## Abstract (arXiv)

We study generalizable policy learning from demonstrations for complex low-level control (e.g., contact-rich object manipulations). We propose a novel hierarchical imitation learning method that utilizes sub-optimal demos. Firstly, we propose an observation space-agnostic approach that efficiently discovers the multi-step subskill decomposition of the demos in an unsupervised manner. By grouping temporarily close and functionally similar actions into subskill-level demo segments, the observations at the segment boundaries constitute a chain of planning steps for the task, which we refer to as the chain-of-thought (CoT). Next, we propose a Transformer-based design that effectively learns to predict the CoT as the subskill-level guidance. We couple action and subskill predictions via learnable prompt tokens and a hybrid masking strategy, which enable dynamically updated guidance at test time and improve feature representation of the trajectory for generalizable policy learning. Our method, Chain-of-Thought Predictive Control (CoTPC), consistently surpasses existing strong baselines on challenging manipulation tasks with sub-optimal demos.