# Watch Your Steps: Observable and Modular Chains of Thought

arXiv: https://arxiv.org/abs/2409.15359 · HTML: https://mlsanity.com/paper/2409.15359/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 17 Sep 2024.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 18 of 18)

- [Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them](https://mlsanity.com/paper/2210.09261/) (2022)
- [Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting](https://mlsanity.com/paper/2305.04388/) (2023)
- [Program of Thoughts Prompting: Disentangling Computation from Reasoning for Numerical Reasoning Tasks](https://mlsanity.com/paper/2211.12588/) (2022)
- [DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelines](https://mlsanity.com/paper/2310.03714/) (2023)
- [Large Language Models](https://mlsanity.com/paper/2307.05782/) (2023)
- [Measuring Faithfulness in Chain-of-Thought Reasoning](https://mlsanity.com/paper/2307.13702/) (2023)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Towards Reasoning in Large Language Models: A Survey](https://mlsanity.com/paper/2212.10403/) (2022)
- [Towards Revealing the Mystery behind Chain of Thought: A Theoretical Perspective](https://mlsanity.com/paper/2305.15408/) (2023)
- [PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization](https://mlsanity.com/paper/2310.16427/) (2023)
- [Chain of Code: Reasoning with a Language Model-Augmented Code Emulator](https://mlsanity.com/paper/2312.04474/) (2023)
- [Why think step by step? Reasoning emerges from the locality of experience](https://mlsanity.com/paper/2304.03843/) (2023)
- [On Measuring Faithfulness or Self-consistency of Natural Language Explanations](https://mlsanity.com/paper/2311.07466/) (2023)
- [Post Hoc Explanations of Language Models Can Improve Language Models](https://mlsanity.com/paper/2305.11426/) (2023)
- [How Likely Do LLMs with CoT Mimic Human Reasoning?](https://mlsanity.com/paper/2402.16048/) (2024)
- [Language Models as Compilers: Simulating Pseudocode Execution Improves Algorithmic Reasoning in Language Models](https://mlsanity.com/paper/2404.02575/) (2024)
- [Understanding the Effect of Noise in LLM Training Data with Algorithmic Chains of Thought](https://mlsanity.com/paper/2402.04004/) (2024)
- [Learning to Reason via Program Generation, Emulation, and Search](https://mlsanity.com/paper/2405.16337/) (2024)

## Built on by (most cited first, 3 of 3)

- [Localizing and Correcting Errors for LLM-based Planners](https://mlsanity.com/paper/2602.00276/) (2026)
- [Semi-structured LLM Reasoners Can Be Rigorously Audited](https://mlsanity.com/paper/2505.24217/) (2025)
- [Inducing Reasoning Primitives from Agent Traces](https://mlsanity.com/paper/2606.02994/) (2026)

## Abstract (arXiv)

We propose a variant of chain of thought (CoT) prompting called Program Trace Prompting that makes explanations more observable while preserving the power, generality and flexibility of CoT. In our approach, few-shot CoT demonstrations are wrapped in a formal syntax based on Python, and each prompt: identifies and names steps; defines the input/output behavior of steps; and replaces CoT explanations of in-context examples with chains of these formalized steps on the same examples. Program Trace Prompting is applicable to many tasks, achieving strong results on the 23 diverse tasks in the BIG-Bench Hard benchmark. More importantly, by instrumenting explanations in this way, we enable new types of analysis. In particular, we identify "non-local errors" (which correspond to incorrectly learning the reasoning method illustrated in the demonstrations) as an unaddressed issue in CoT learning, and we present methods for verifying the modularity of steps in a CoT explanation.