# Analyzing Chain-of-Thought Prompting in Large Language Models via Gradient-based Feature Attributions

arXiv: https://arxiv.org/abs/2307.13339 · HTML: https://mlsanity.com/paper/2307.13339/

## Key facts

- Cited by 32 papers in the map (data through 14 Oct 2026).
- Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning.
- Builds on 10 of its 25 references that are in the map.
- First posted on arXiv 25 Jul 2023.

## Placed in

- LLM reasoning / [Prompting and chain-of-thought reasoning](https://mlsanity.com/lr/prompting-and-chain-of-thought-reasoning/)

## Builds on (most cited first, 10 of 10)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Automatic Chain of Thought Prompting in Large Language Models](https://mlsanity.com/paper/2210.03493/) (2022)
- [Can language models learn from explanations in context?](https://mlsanity.com/paper/2204.02329/) (2022)
- [Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters](https://mlsanity.com/paper/2212.10001/) (2022)
- [Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango](https://mlsanity.com/paper/2209.07686/) (2022)
- [Interpreting Language Models with Contrastive Explanations](https://mlsanity.com/paper/2202.10419/) (2022)
- [AllenNLP Interpret: A Framework for Explaining Predictions of NLP Models](https://mlsanity.com/paper/1909.09251/) (2019)

## Built on by (most cited first, 30 of 32)

- [Explainability for Large Language Models: A Survey](https://mlsanity.com/paper/2309.01029/) (2023)
- [Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future](https://mlsanity.com/paper/2309.15402/) (2023)
- [The Impact of Reasoning Step Length on Large Language Models](https://mlsanity.com/paper/2401.04925/) (2024)
- [Thought Anchors: Which LLM Reasoning Steps Matter?](https://mlsanity.com/paper/2506.19143/) (2025)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [A Survey on Large Language Models for Mathematical Reasoning](https://mlsanity.com/paper/2506.08446/) (2025)
- [Explainable Artificial Intelligence: A Survey of Needs, Techniques, Applications, and Future Direction](https://mlsanity.com/paper/2409.00265/) (2024)
- [Transformers Learn to Implement Multi-step Gradient Descent with Chain of Thought](https://mlsanity.com/paper/2502.21212/) (2025)
- [Unveiling the Statistical Foundations of Chain-of-Thought Prompting Methods](https://mlsanity.com/paper/2408.14511/) (2024)
- [Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning](https://mlsanity.com/paper/2407.01687/) (2024)
- [Locate, Steer, and Improve: A Practical Survey of Actionable Mechanistic Interpretability in Large Language Models](https://mlsanity.com/paper/2601.14004/) (2026)
- [Stop Reasoning! When Multimodal LLM with Chain-of-Thought Reasoning Meets Adversarial Image](https://mlsanity.com/paper/2402.14899/) (2024)
- [Landscape of Thoughts: Visualizing the Reasoning Process of Large Language Models](https://mlsanity.com/paper/2503.22165/) (2025)
- [The Sound of Healthcare: Improving Medical Transcription ASR Accuracy with Large Language Models](https://mlsanity.com/paper/2402.07658/) (2024)
- [Explainable Chain-of-Thought Reasoning: An Empirical Analysis on State-Aware Reasoning Dynamics](https://mlsanity.com/paper/2509.00190/) (2025)
- [A Theoretical Understanding of Chain-of-Thought: Coherent Reasoning and Error-Aware Demonstration](https://mlsanity.com/paper/2410.16540/) (2024)
- [VLM's Eye Examination: Instruct and Inspect Visual Competency of Vision Language Models](https://mlsanity.com/paper/2409.14759/) (2024)
- [Chain-of-Thought in Large Language Models: Decoding, Projection, and Activation](https://mlsanity.com/paper/2412.03944/) (2024)
- [Caught in the Act: a mechanistic approach to detecting deception](https://mlsanity.com/paper/2508.19505/) (2025)
- [Do LLMs Encode Functional Importance of Reasoning Tokens?](https://mlsanity.com/paper/2601.03066/) (2026)
- [Securing the Future of GenAI: Policy and Technology](https://mlsanity.com/paper/2407.12999/) (2024)
- [Towards detecting unanticipated bias in Large Language Models](https://mlsanity.com/paper/2404.02650/) (2024)
- [Brain-like Functional Organization within Large Language Models](https://mlsanity.com/paper/2410.19542/) (2024)
- [Which Words Matter Most in Zero-Shot Prompts?](https://mlsanity.com/paper/2502.03418/) (2025)
- [XiHeFusion: Harnessing Large Language Models for Science Communication in Nuclear Fusion](https://mlsanity.com/paper/2502.05615/) (2025)
- [Can Large Language Models Simulate Human Responses? A Case Study of Stated Preference Experiments in the Context of Heating-related Choices](https://mlsanity.com/paper/2503.10652/) (2025)
- [The Potential of CoT for Reasoning: A Closer Look at Trace Dynamics](https://mlsanity.com/paper/2602.14903/) (2026)
- [Reasoning Fails Where Step Flow Breaks](https://mlsanity.com/paper/2604.06695/) (2026)
- [How Chain-of-Thought Works? Tracing Information Flow from Decoding, Projection, and Activation](https://mlsanity.com/paper/2507.20758/) (2025)
- [What Really Counts? Examining Step and Token Level Attribution in Multilingual CoT Reasoning](https://mlsanity.com/paper/2511.15886/) (2025)

## Abstract (arXiv)

Chain-of-thought (CoT) prompting has been shown to empirically improve the accuracy of large language models (LLMs) on various question answering tasks. While understanding why CoT prompting is effective is crucial to ensuring that this phenomenon is a consequence of desired model behavior, little work has addressed this; nonetheless, such an understanding is a critical prerequisite for responsible model deployment. We address this question by leveraging gradient-based feature attribution methods which produce saliency scores that capture the influence of input tokens on model output. Specifically, we probe several open-source LLMs to investigate whether CoT prompting affects the relative importances they assign to particular input tokens. Our results indicate that while CoT prompting does not increase the magnitude of saliency scores attributed to semantically relevant tokens in the prompt compared to standard few-shot prompting, it increases the robustness of saliency scores to question perturbations and variations in model output.