# Chain of Thought Imitation with Procedure Cloning

arXiv: https://arxiv.org/abs/2205.10816 · HTML: https://mlsanity.com/paper/2205.10816/

## Key facts

- Cited by 30 papers in the map (data through 14 Oct 2026).
- Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning.
- Builds on 57 of its 102 references that are in the map.
- First posted on arXiv 22 May 2022.

## Placed in

- LLM reasoning / [Spatial and embodied reasoning](https://mlsanity.com/lr/spatial-and-embodied-reasoning/)
- Offline RL and imitation learning from data / [Imitation learning methods](https://mlsanity.com/offlinerl/imitation-learning-methods/)

## Builds on (most cited first, 30 of 57)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [D4RL: Datasets for Deep Data-Driven Reinforcement Learning](https://mlsanity.com/paper/2004.07219/) (2020)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Show Your Work: Scratchpads for Intermediate Computation with Language Models](https://mlsanity.com/paper/2112.00114/) (2021)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problems](https://mlsanity.com/paper/1705.04146/) (2017)
- [Offline Reinforcement Learning as One Big Sequence Modeling Problem](https://mlsanity.com/paper/2106.02039/) (2021)
- [Learning Quadrupedal Locomotion over Challenging Terrain](https://mlsanity.com/paper/2010.11251/) (2020)
- [Minimax Regret Bounds for Reinforcement Learning](https://mlsanity.com/paper/1703.05449/) (2017)
- [Leveraging Procedural Generation to Benchmark Reinforcement Learning](https://mlsanity.com/paper/1912.01588/) (2019)
- [Implicit Behavioral Cloning](https://mlsanity.com/paper/2109.00137/) (2021)
- [Learning Invariant Representations for Reinforcement Learning without Reconstruction](https://mlsanity.com/paper/2006.10742/) (2020)
- [Reinforcement Learning with Unsupervised Auxiliary Tasks](https://mlsanity.com/paper/1611.05397/) (2016)
- [Learning by Cheating](https://mlsanity.com/paper/1912.12294/) (2019)
- [Explain Yourself! Leveraging Language Models for Commonsense Reasoning](https://mlsanity.com/paper/1906.02361/) (2019)
- [Quantifying Generalization in Reinforcement Learning](https://mlsanity.com/paper/1812.02341/) (2018)
- [Stabilizing Transformers for Reinforcement Learning](https://mlsanity.com/paper/1910.06764/) (2019)
- [Transformers as Soft Reasoners over Language](https://mlsanity.com/paper/2002.05867/) (2020)
- [Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agents](https://mlsanity.com/paper/1709.06009/) (2017)
- [Taming the Monster: A Fast and Simple Algorithm for Contextual Bandits](https://mlsanity.com/paper/1402.0555/) (2014)
- [Learning to Navigate in Complex Environments](https://mlsanity.com/paper/1611.03673/) (2016)
- [Online Decision Transformer](https://mlsanity.com/paper/2202.05607/) (2022)
- [Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning](https://mlsanity.com/paper/1703.07710/) (2017)
- [A Study on Overfitting in Deep Reinforcement Learning](https://mlsanity.com/paper/1804.06893/) (2018)

## Built on by (most cited first, 30 of 30)

- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [Foundation Models for Decision Making: Problems, Methods, and Opportunities](https://mlsanity.com/paper/2303.04129/) (2023)
- [Stream of Search (SoS): Learning to Search in Language](https://mlsanity.com/paper/2404.03683/) (2024)
- [Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thought](https://mlsanity.com/paper/2501.04682/) (2025)
- [Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrapping](https://mlsanity.com/paper/2402.14083/) (2024)
- [Video as the New Language for Real-World Decision Making](https://mlsanity.com/paper/2402.17139/) (2024)
- [Towards Better Chain-of-Thought Prompting Strategies: A Survey](https://mlsanity.com/paper/2310.04959/) (2023)
- [A Survey on Large Language Models for Mathematical Reasoning](https://mlsanity.com/paper/2506.08446/) (2025)
- [A Survey on Transformers in Reinforcement Learning](https://mlsanity.com/paper/2301.03044/) (2023)
- [System-1.x: Learning to Balance Fast and Slow Planning with Language Models](https://mlsanity.com/paper/2407.14414/) (2024)
- [Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agents](https://mlsanity.com/paper/2509.03581/) (2025)
- [Chain-of-Thought Predictive Control](https://mlsanity.com/paper/2304.00776/) (2023)
- [CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Grounding](https://mlsanity.com/paper/2310.06214/) (2023)
- [A Theory for Length Generalization in Learning to Reason](https://mlsanity.com/paper/2404.00560/) (2024)
- [Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexity](https://mlsanity.com/paper/2509.07339/) (2025)
- [Models That Prove Their Own Correctness](https://mlsanity.com/paper/2405.15722/) (2024)
- [FLARE: Faithful Logic-Aided Reasoning and Exploration](https://mlsanity.com/paper/2410.11900/) (2024)
- [RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMs](https://mlsanity.com/paper/2505.13697/) (2025)
- [On Learning Verifiers and Implications to Chain-of-Thought Reasoning](https://mlsanity.com/paper/2505.22650/) (2025)
- [Multi-Environment Pretraining Enables Transfer to Action Limited Datasets](https://mlsanity.com/paper/2211.13337/) (2022)
- [HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Model](https://mlsanity.com/paper/2406.01882/) (2024)
- [Learning to Better Search with Language Models via Guided Reinforced Self-Training](https://mlsanity.com/paper/2410.02992/) (2024)
- [Offline Learning and Forgetting for Reasoning with Large Language Models](https://mlsanity.com/paper/2504.11364/) (2025)
- [Toward Open-ended Embodied Tasks Solving](https://mlsanity.com/paper/2312.05822/) (2023)
- [InfoCon: Concept Discovery with Generative and Discriminative Informativeness](https://mlsanity.com/paper/2404.10606/) (2024)
- [Masked Distillation: Internalizing the Chain-of-Thought in Language Models](https://mlsanity.com/paper/2607.22629/) (2026)
- [Replacing thinking with tool usage enables reasoning in small language models](https://mlsanity.com/paper/2507.05065/) (2025)
- [Draw it like Euclid: Teaching transformer models to generate CAD profiles using ruler and compass construction steps](https://mlsanity.com/paper/2601.09428/) (2026)
- [TSLM: Tree-Structured Language Modeling for Divergent Thinking](https://mlsanity.com/paper/2601.22688/) (2026)
- [LaRec: Unleashing LLM-based Latent Reasoning for Generative Recommendation](https://mlsanity.com/paper/2607.24617/) (2026)

## Abstract (arXiv)

Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning problem in which one fits a function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning as a supervised input-output learning problem allows for applicability in a wide variety of settings, it is also an overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstrations via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they are nevertheless informative as a way to explain an expert's mapping of state to actions. To properly leverage expert procedure information without relying on the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert's procedure directly is infeasible.