# Short-circuiting Shortcuts: Mechanistic Investigation of Shortcuts in Text Classification

arXiv: https://arxiv.org/abs/2505.06032 · HTML: https://mlsanity.com/paper/2505.06032/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 44 references that are in the map.
- First posted on arXiv 9 May 2025.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [Generating Natural Language Adversarial Examples](https://mlsanity.com/paper/1804.07998/) (2018)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Stress Test Evaluation for Natural Language Inference](https://mlsanity.com/paper/1806.00692/) (2018)
- [Post-hoc Interpretability for Neural NLP: A Survey](https://mlsanity.com/paper/2108.04840/) (2021)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Explaining NLP Models via Minimal Contrastive Editing (MiCE)](https://mlsanity.com/paper/2012.13985/) (2020)
- [Towards Interpreting and Mitigating Shortcut Learning Behavior of NLU Models](https://mlsanity.com/paper/2103.06922/) (2021)
- [Identifying and Mitigating Spurious Correlations for Improving Robustness in NLP Models](https://mlsanity.com/paper/2110.07736/) (2021)
- ["Will You Find These Shortcuts?" A Protocol for Evaluating the Faithfulness of Input Salience Methods for Text Classification](https://mlsanity.com/paper/2111.07367/) (2021)
- [Why Attentions May Not Be Interpretable?](https://mlsanity.com/paper/2006.05656/) (2020)
- [Combining Feature and Instance Attribution to Detect Artifacts](https://mlsanity.com/paper/2107.00323/) (2021)
- [Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinations](https://mlsanity.com/paper/2403.18167/) (2024)
- [An Empirical Comparison of Instance Attribution Methods for NLP](https://mlsanity.com/paper/2104.04128/) (2021)
- [Finding Dataset Shortcuts with Grammar Induction](https://mlsanity.com/paper/2210.11560/) (2022)
- [The Role of Syntactic Span Preferences in Post-Hoc Explanation Disagreement](https://mlsanity.com/paper/2403.19424/) (2024)

## Built on by (most cited first, 5 of 5)

- [CoSPlan: Corrective Sequential Planning via Scene Graph Incremental Updates](https://mlsanity.com/paper/2512.10342/) (2025)
- [Judge Circuits Explain Format-Induced Inconsistency in LLM-as-a-Judge](https://mlsanity.com/paper/2605.16023/) (2026)
- [Explanation Bias is a Product: Revealing the Hidden Lexical and Position Preferences in Post-Hoc Feature Attribution](https://mlsanity.com/paper/2512.11108/) (2025)
- [Calibration vs Decision Making: Revisiting the Reliability Paradox in Unlearned Language Models](https://mlsanity.com/paper/2605.20915/) (2026)
- [A Polyphonic Conception of AI Understanding](https://mlsanity.com/paper/2609.36079/) (2026)

## Abstract (arXiv)

Reliance on spurious correlations (shortcuts) has been shown to underlie many of the successes of language models. Previous work focused on identifying the input elements that impact prediction. We investigate how shortcuts are actually processed within the model's decision-making mechanism. We use actor names in movie reviews as controllable shortcuts with known impact on the outcome. We use mechanistic interpretability methods and identify specific attention heads that focus on shortcuts. These heads gear the model towards a label before processing the complete input, effectively making premature decisions that bypass contextual analysis. Based on these findings, we introduce Head-based Token Attribution (HTA), which traces intermediate decisions back to input tokens. We show that HTA is effective in detecting shortcuts in LLMs and enables targeted mitigation by selectively deactivating shortcut-related attention heads.