mlsanityData to 7 Oct 2026

Overthinking the Truth: Understanding how Language Models Process False Demonstrations

arXiv 2307.09476 ↗First posted 18 Jul 2023cs.LG cs.AI cs.CLcited by 78

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Modern language models can imitate complex patterns through few-shot learning, enabling them to complete challenging tasks without fine-tuning. However, imitation can also lead models to reproduce inaccuracies or harmful content if present in the context. We study harmful imitation through the lens of a model's internal representations, and identify two related phenomena: "overthinking" and "false induction heads". The first phenomenon, overthinking, appears when we decode predictions from intermediate layers, given correct vs. incorrect few-shot demonstrations. At early layers, both demonstrations induce similar model behavior, but the behavior diverges sharply at some "critical layer", after which the accuracy given incorrect demonstrations progressively decreases. The second phenomenon, false induction heads, are a possible mechanistic cause of overthinking: these are heads in late layers that attend to and copy false information from previous demonstrations, and whose ablation reduces overthinking. Beyond scientific understanding, our results suggest that studying intermediate model computations could be a promising avenue for understanding and guarding against harmful model behaviors.

Key facts. Cited by 78 papers in the map (data through 7 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 39 of its 58 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

39 of 58 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?25 Feb

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  12. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
26 more
  1. 2022Mass-Editing Memory in a Transformercited by 976
  2. 2022In-context Learning and Induction Headscited by 930
  3. 2021An Explanation of In-context Learning as Implicit Bayesian Inferencecited by 915
  4. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  5. 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 760
  6. 2022GPT-NeoX-20B: An Open-Source Autoregressive Language Modelcited by 711
  7. 2019Designing and Interpreting Probes with Control Taskscited by 608
  8. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  9. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  10. 2020TweetEval: Unified Benchmark and Comparative Evaluation for Tweet Classificationcited by 414
  11. 2021Causal Abstractions of Neural Networkscited by 401
  12. 2023Larger language models do in-context learning differentlycited by 386
  13. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  14. 2013Good Debt or Bad Debt: Detecting Semantic Orientations in Economic Textscited by 310
  15. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 267
  16. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  17. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  18. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  19. 2022Ground-Truth Labels Matter: A Deeper Look into Input-Label Demonstrationscited by 116
  20. 2022Capturing Failures of Large Language Models via Human Cognitive Biasescited by 97
  21. 2015Conditional Deep Learning for Energy-Efficient and Enhanced Pattern Recognitioncited by 67
  22. 2022Evaluating the Susceptibility of Pre-Trained Language Models via Handcrafted Adversarial Examplescited by 65
  23. 2020Effective Transfer Learning for Identifying Similar Questions: Matching User Questions to COVID-19 FAQscited by 48
  24. 2020Investigating Societal Biases in a Poetry Composition Systemcited by 47
  25. 2017Probabilistic Adaptive Computation Timecited by 4
  26. 2022Understanding the Robustness of Multi-Exit Models under Common Corruptionscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 27; 2025: 22; 2026: 20.

78 papers.

Sort:

By area:

  1. 2023Function Vectors in Large Language Modelscited by 292
  2. 2024Personal LLM Agents: Insights and Survey about the Capability, Efficiency and Securitycited by 257
  3. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  4. 2024Devils in Middle Layers of Large Vision-Language Models: Interpreting, Detecting and Mitigating Object Hallucinations via Attention Lenscited by 139
  5. 2024Interpreting and Editing Vision-Language Representations to Mitigate Hallucinationscited by 103
  6. 2024Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and Mitigating Knowledge Conflicts in Language Modelscited by 87
  7. 2023Cognitive Mirage: A Review of Hallucinations in Large Language Modelscited by 83
  8. 2024Covert Malicious Finetuning: Challenges in Safeguarding LLM Adaptationcited by 74
  9. 2023Copy Suppression: Comprehensively Understanding an Attention Headcited by 73
  10. 2023Attention Satisfies: A Constraint-Satisfaction Lens on Factual Errors of Language Modelscited by 67
  11. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  12. 2024Get my drift? Catching LLM Task Drift with Activation Deltascited by 61
38 more
  1. 2024From Understanding to Utilization: A Survey on Explainability for Large Language Modelscited by 47
  2. 2024In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigationcited by 43
  3. 2024On Large Language Models' Hallucination with Regard to Known Factscited by 43
  4. 2024Understanding the Dark Side of LLMs' Intrinsic Self-Correctioncited by 42
  5. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
  6. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  7. 2024Emergence of Hidden Capabilities: Exploring Learning Dynamics in Concept Spacecited by 33
  8. 2024Out-of-distribution generalization via composition: a lens through induction heads in Transformerscited by 28
  9. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 27
  10. 2023Localizing Lying in Llama: Understanding Instructed Dishonesty on True-False Questions Through Prompting, Probing, and Patchingcited by 24
  11. 2023DecoderLens: Layerwise Interpretation of Encoder-Decoder Transformerscited by 22
  12. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  13. 2024Noise Injection Reveals Hidden Capabilities of Sandbagging Language Modelscited by 22
  14. 2024Optimal ablation for interpretabilitycited by 21
  15. 2024Jailbreak Antidote: Runtime Safety-Utility Balance via Sparse Representation Adjustment in Large Language Modelscited by 21
  16. 2025SelfElicit: Your Language Model Secretly Knows Where is the Relevant Evidencecited by 18
  17. 2025Probing the Geometry of Truth: Consistency and Generalization of Truth Directions in LLMs Across Logical Transformations and Question Answering Taskscited by 18
  18. 2025CER: Confidence Enhanced Reasoning in LLMscited by 16
  19. 2024Anchored Answers: Unravelling Positional Bias in GPT-2's Multiple-Choice Questionscited by 15
  20. 2025Empowering GraphRAG with Knowledge Filtering and Integrationcited by 15
  21. 2025Safety Reasoning with Guidelinescited by 14
  22. 2024Looking Beyond The Top-1: Transformers Determine Top Tokens In Ordercited by 12
  23. 2025Learning Task Representations from In-Context Learningcited by 11
  24. 2025When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Modelscited by 11
  25. 2025Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledgecited by 10
  26. 2025Unifying Attention Heads and Task Vectors via Hidden State Geometry in In-Context Learningcited by 10
  27. 2025Interpretation Meets Safety: A Survey on Interpretation Methods and Tools for Improving LLM Safetycited by 10
  28. 2024Unraveling the Mechanics of Learning-Based Demonstration Selection for In-Context Learningcited by 9
  29. 2025Improve Decoding Factuality by Token-wise Cross Layer Entropy of Large Language Modelscited by 7
  30. 2025HICD: Hallucination-Inducing via Attention Dispersion for Contrastive Decoding to Mitigate Hallucinations in Large Language Modelscited by 7
  31. 2026LLMs Know They're Wrong and Agree Anyway: The Shared Sycophancy-Lying Circuitcited by 7
  32. 2023LLM Factoscope: Uncovering LLMs' Factual Discernment through Inner States Analysiscited by 6
  33. 2024Multilevel Interpretability Of Artificial Neural Networks: Leveraging Framework And Methods From Neurosciencecited by 5
  34. 2025Revealing the Deceptiveness of Knowledge Editing: A Mechanistic Analysis of Superficial Editingcited by 5
  35. 2026Mitigating Conversational Inertia in Multi-Turn Agentscited by 5
  36. 2024Self-Demos: Eliciting Out-of-Demonstration Generalizability in Large Language Modelscited by 4
  37. 2025DAST: Difficulty-Aware Self-Training on Large Language Modelscited by 3
  38. 2025Languages in Whisper-Style Speech Encoders Align Both Phonetically and Semanticallycited by 3

50 most cited of 78 All 78 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model
  2. 2026Overthinking Causes Hallucination: Tracing Confounder Propagation in Vision Language Models
  3. 2026Probing the Misaligned Thinking Process of Language Models
  4. 2025Layer of Truth: Probing Belief Shifts under Continual Pre-Training Poisoning
  5. 2024Unfamiliar Finetuning Examples Control How Language Models Hallucinate
  6. 2024Brittle Minds, Fixable Activations: Understanding Belief Representations in Language Models
  7. 2026When Decodability Is Not Enough: Logical Validity Representations, Behavioral Dissociation, and Causal Tests in Language Models
  8. 2025The Curious Case of Factuality Finetuning: Models' Internal Beliefs Can Improve Factuality