The Readout Shortcut: Positional Number Copying Dominates Arithmetic CoT Readout in Small Language Models
On the map: LLM reasoning → Prompting and chain-of-thought reasoning
Abstract (arXiv)
Chain-of-thought (CoT) prompting is necessary for arithmetic in small language models, yet shuffling its steps preserves most performance. What does CoT contribute if not logical sequencing? In three 1-3B instruction-tuned LMs on GSM8K, we isolate the answer-readout stage via prefix completion and identify a positional shortcut: the model copies whichever number occupies the trailing position before the answer delimiter, regardless of intermediate reasoning. Gold-answer presence accounts for 54-92 pp of accuracy (89-92% of each model's teacher-forcing ceiling); even on incorrect items, the final answer matches the last CoT number 95-96% of the time. The copy channel takes precedence over retained-context completion: replacing the trailing number with a wrong value collapses accuracy to near-zero despite correct intermediates, yet removing it recovers 5-32 pp above that floor--even single-step arithmetic the model can otherwise perform is suppressed when a copyable number is present. Qwen and Llama copy novel distractors 87-95% of the time; Gemma gates selectively. Head-level ablation implicates architecture-specific head sets; the effect replicates on GSM-Symbolic. On non-arithmetic BBH tasks, shuffle retention drops sharply; at 7-8B, content-selective gating emerges. Step-level faithfulness evaluations risk conflating positional answer transport with genuine computation--a failure mode for CoT-based oversight.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 25 of its 33 references that are in the map. First posted on arXiv 20 May 2026.
Builds on
25 of 33 references.
Closest ancestors · 10
Same direction, newest first
- 2026Rethinking Dense Sequential Chains: Reasoning Language Models Can Extract Answers from Sparse, Order-Shuffling Chain-of-Thoughts8 May
- 2025Chain of Thought Monitorability: A New and Fragile Opportunity for AI Safety15 Jul
- 2025Thought Anchors: Which LLM Reasoning Steps Matter?23 Jun
- 2025Reasoning Models Don't Always Say What They Think8 May
- 2024Let's Think Dot by Dot: Hidden Computation in Transformer Language Models24 Apr
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Prompting7 May
- 2022Towards Understanding Chain-of-Thought Prompting: An Empirical Study of What Matters20 Dec
- 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Them17 Oct
- 2022Text and Patterns: For Effective Chain of Thought, It Takes Two to Tango16 Sep
- 2022Large Language Models are Zero-Shot Reasoners24 May
Other references · 15
- 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
- 2023Let's Verify Step by Stepcited by 3,789
- 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
- 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,998
- 2020Shortcut Learning in Deep Neural Networkscited by 1,969
- 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,346
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
- 2022In-context Learning and Induction Headscited by 964
- 2021Are NLP Models really able to Solve Simple Math Word Problems?cited by 878
- 2019Right for the Wrong Reasons: Diagnosing Syntactic Heuristics in Natural Language Inferencecited by 876
- 2023Large Language Models Can Be Easily Distracted by Irrelevant Contextcited by 841
- 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 359
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2024Break the Chain: Large Language Models Can be Shortcut Reasoners
- 2024LLMs Faithfully and Iteratively Compute Answers During CoT: A Systematic Analysis With Multi-step Arithmetics
- 2026Soft Guidance Starts to Outperform CoT Prompting as LLMs Improve
- 2025When Thinking Fails: The Pitfalls of Reasoning for Instruction-Following in LLMs
- 2025Unspoken Hints: Accuracy Without Acknowledgement in LLM Reasoning
- 2024Deciphering the Factors Influencing the Efficacy of Chain-of-Thought: Probability, Memorization, and Noisy Reasoning
- 2023Stress Testing Chain-of-Thought Prompting for Large Language Models
- 2026The Potential of CoT for Reasoning: A Closer Look at Trace Dynamics