mlsanityData to 14 Oct 2026

Chain of Thought Imitation with Procedure Cloning

arXiv 2205.10816 ↗First posted 22 May 2022cs.LG cs.AIcited by 30

On the map: LLM reasoning → Spatial and embodied reasoning

also in 1 more
Abstract (arXiv)

Imitation learning aims to extract high-performance policies from logged demonstrations of expert behavior. It is common to frame imitation learning as a supervised learning problem in which one fits a function approximator to the input-output mapping exhibited by the logged demonstrations (input observations to output actions). While the framing of imitation learning as a supervised input-output learning problem allows for applicability in a wide variety of settings, it is also an overly simplistic view of the problem in situations where the expert demonstrations provide much richer insight into expert behavior. For example, applications such as path navigation, robot manipulation, and strategy games acquire expert demonstrations via planning, search, or some other multi-step algorithm, revealing not just the output action to be imitated but also the procedure for how to determine this action. While these intermediate computations may use tools not available to the agent during inference (e.g., environment simulators), they are nevertheless informative as a way to explain an expert's mapping of state to actions. To properly leverage expert procedure information without relying on the privileged tools the expert may have used to perform the procedure, we propose procedure cloning, which applies supervised sequence prediction to imitate the series of expert computations. This way, procedure cloning learns not only what to do (i.e., the output action), but how and why to do it (i.e., the procedure). Through empirical analysis on navigation, simulated robotic manipulation, and game-playing environments, we show that imitating the intermediate computations of an expert's behavior enables procedure cloning to learn policies exhibiting significant generalization to unseen environment configurations, including those configurations for which running the expert's procedure directly is infeasible.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in Spatial and embodied reasoning, one of 17 directions in LLM reasoning. Builds on 57 of its 102 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

57 of 102 references.

References · 57

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Scaling Laws for Neural Language Modelscited by 6,586
  6. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  7. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2022STaR: Bootstrapping Reasoning With Reasoningcited by 940
  10. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  11. 2017Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithmcited by 734
  12. 2017Program Induction by Rationale Generation : Learning to Solve and Explain Algebraic Word Problemscited by 650
38 more
  1. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  2. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  3. 2017Minimax Regret Bounds for Reinforcement Learningcited by 503
  4. 2019Leveraging Procedural Generation to Benchmark Reinforcement Learningcited by 483
  5. 2021Implicit Behavioral Cloningcited by 452
  6. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  7. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 404
  8. 2019Learning by Cheatingcited by 383
  9. 2019Explain Yourself! Leveraging Language Models for Commonsense Reasoningcited by 367
  10. 2018Quantifying Generalization in Reinforcement Learningcited by 352
  11. 2019Stabilizing Transformers for Reinforcement Learningcited by 270
  12. 2020Transformers as Soft Reasoners over Languagecited by 270
  13. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  14. 2014Taming the Monster: A Fast and Simple Algorithm for Contextual Banditscited by 201
  15. 2016Learning to Navigate in Complex Environmentscited by 198
  16. 2022Online Decision Transformercited by 196
  17. 2017Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learningcited by 176
  18. 2018A Study on Overfitting in Deep Reinforcement Learningcited by 174
  19. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  20. 2019Goal-conditioned Imitation Learningcited by 162
  21. 2016Value Iteration Networkscited by 160
  22. 2020Acme: A Research Framework for Distributed Reinforcement Learningcited by 146
  23. 2018Reinforcement and Imitation Learning for Diverse Visuomotor Skillscited by 126
  24. 2018Generalization and Regularization in DQNcited by 118
  25. 2021Generalized Decision Transformer for Offline Hindsight Information Matchingcited by 106
  26. 2019Network Randomization: A Simple Technique for Generalization in Deep Reinforcement Learningcited by 104
  27. 2015Neural Programmer-Interpreterscited by 101
  28. 2016Playing FPS Games with Deep Reinforcement Learningcited by 95
  29. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 91
  30. 2018Hierarchical Imitation and Reinforcement Learningcited by 88
  31. 2019IRIS: Implicit Reinforcement without Interaction at Scale for Learning Control from Offline Robot Manipulation Datacited by 86
  32. 2015Massively Parallel Methods for Deep Reinforcement Learningcited by 82
  33. 2022Can Wikipedia Help Offline Reinforcement Learning?cited by 77
  34. 2017Neural Task Programming: Learning to Generalize Across Hierarchical Taskscited by 76
  35. 2021Visual Adversarial Imitation Learning using Variational Modelscited by 42
  36. 2020Provable Representation Learning for Imitation Learning via Bi-level Optimizationcited by 41
  37. 2021TRAIL: Near-Optimal Imitation Learning with Suboptimal Datacited by 39
  38. 2021Hierarchical Few-Shot Imitation with Skill Transition Modelscited by 37

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 1; 2023: 7; 2024: 10; 2025: 8; 2026: 4.

30 papers.

Sort:

By area:

  1. 2023Language Models can Solve Computer Taskscited by 372
  2. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 164
  3. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  4. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  5. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  6. 2024Video as the New Language for Real-World Decision Makingcited by 87
  7. 2023Towards Better Chain-of-Thought Prompting Strategies: A Surveycited by 62
  8. 2025A Survey on Large Language Models for Mathematical Reasoningcited by 52
  9. 2023A Survey on Transformers in Reinforcement Learningcited by 50
  10. 2024System-1.x: Learning to Balance Fast and Slow Planning with Language Modelscited by 35
  11. 2025Learning When to Plan: Efficiently Allocating Test-Time Compute for LLM Agentscited by 20
  12. 2023Chain-of-Thought Predictive Controlcited by 18
18 more
  1. 2023CoT3DRef: Chain-of-Thoughts Data-Efficient 3D Visual Groundingcited by 13
  2. 2024A Theory for Length Generalization in Learning to Reasoncited by 11
  3. 2025Performative Thinking? The Brittle Correlation Between CoT Length and Problem Complexitycited by 9
  4. 2024Models That Prove Their Own Correctnesscited by 7
  5. 2024FLARE: Faithful Logic-Aided Reasoning and Explorationcited by 7
  6. 2025RL in Name Only? Analyzing the Structural Assumptions in RL post-training for LLMscited by 7
  7. 2025On Learning Verifiers and Implications to Chain-of-Thought Reasoningcited by 6
  8. 2022Multi-Environment Pretraining Enables Transfer to Action Limited Datasetscited by 5
  9. 2024HoneyGPT: Breaking the Trilemma in Terminal Honeypots with Large Language Modelcited by 5
  10. 2024Learning to Better Search with Language Models via Guided Reinforced Self-Trainingcited by 5
  11. 2025Offline Learning and Forgetting for Reasoning with Large Language Modelscited by 4
  12. 2023Toward Open-ended Embodied Tasks Solvingcited by 3
  13. 2024InfoCon: Concept Discovery with Generative and Discriminative Informativenesscited by 3
  14. 2026Masked Distillation: Internalizing the Chain-of-Thought in Language Modelscited by 3
  15. 2025Replacing thinking with tool usage enables reasoning in small language modelscited by 1
  16. 2026Draw it like Euclid: Teaching transformer models to generate CAD profiles using ruler and compass construction stepscited by 1
  17. 2026TSLM: Tree-Structured Language Modeling for Divergent Thinkingcited by 0
  18. 2026LaRec: Unleashing LLM-based Latent Reasoning for Generative Recommendationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
  2. 2024Interactive and Hybrid Imitation Learning: Provably Beating Behavior Cloning
  3. 2025A Smooth Sea Never Made a Skilled SAILOR: Robust Imitation via Learning to Search
  4. 2024Imitation Learning from Suboptimal Demonstrations via Meta-Learning An Action Ranker
  5. 2023Self-Supervised Adversarial Imitation Learning
  6. 2020Concurrent Training Improves the Performance of Behavioral Cloning from Observation
  7. 2018Behavioral Cloning from Observation
  8. 2023Diffusion Model-Augmented Behavioral Cloning