mlsanityData to 7 Oct 2026

Confidence Regulation Neurons in Language Models

arXiv 2406.16254 ↗First posted 24 Jun 2024cs.LG cs.AI cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Despite their widespread use, the mechanisms by which large language models (LLMs) represent and regulate uncertainty in next-token predictions remain largely unexplored. This study investigates two critical components believed to influence this uncertainty: the recently discovered entropy neurons and a new set of components that we term token frequency neurons. Entropy neurons are characterized by an unusually high weight norm and influence the final layer normalization (LayerNorm) scale to effectively scale down the logits. Our work shows that entropy neurons operate by writing onto an unembedding null space, allowing them to impact the residual stream norm with minimal direct effect on the logits themselves. We observe the presence of entropy neurons across a range of models, up to 7 billion parameters. On the other hand, token frequency neurons, which we discover and describe here for the first time, boost or suppress each token's logit proportionally to its log frequency, thereby shifting the output distribution towards or away from the unigram distribution. Finally, we present a detailed case study where entropy neurons actively manage confidence in the setting of induction, i.e. detecting and continuing repeated subsequences.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 55 of its 79 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

55 of 79 references.

References · 55

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  6. 2016Layer Normalizationcited by 5,579
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  11. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  12. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
38 more
  1. 2017Decoupled Weight Decay Regularizationcited by 1,722
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2019Root Mean Square Layer Normalizationcited by 1,425
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  5. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  10. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  13. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  14. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  15. 2021A Survey of Uncertainty in Deep Neural Networkscited by 675
  16. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  17. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  18. 2021Knowledge Neurons in Pretrained Transformerscited by 590
  19. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  20. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  21. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  22. 2021Unsolved Problems in ML Safetycited by 291
  23. 2013Direct and Indirect Effectscited by 276
  24. 2022Prompting GPT-3 To Be Reliablecited by 258
  25. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  26. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  27. 2015Visualizing and Understanding Recurrent Networkscited by 158
  28. 2017Learning to Generate Reviews and Discovering Sentimentcited by 147
  29. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  30. 2022Analyzing Transformers in Embedding Spacecited by 131
  31. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  32. 2023Decomposing Uncertainty for Large Language Models through Input Clarification Ensemblingcited by 113
  33. 2021Transformer visualization via dictionary learning: contextualized embedding as a linear superposition of transformer factorscited by 106
  34. 2024Universal Neurons in GPT2 Language Modelscited by 100
  35. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  36. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  37. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  38. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 26; 2026: 25.

57 papers.

Sort:

By area:

  1. 2024Turning Up the Heat: Min-p Sampling for Creative and Coherent LLM Outputscited by 148
  2. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  3. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  4. 2026How do LLMs Compute Verbal Confidencecited by 23
  5. 2025Transferring Linear Features Across Language Models With Model Stitchingcited by 17
  6. 2025Dense SAE Latents Are Features, Not Bugscited by 17
  7. 2025How Post-Training Reshapes LLMs: A Mechanistic View on Knowledge, Truthfulness, Refusal, and Confidencecited by 12
  8. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  9. 2024You can remove GPT2's LayerNorm by fine-tuningcited by 11
  10. 2024Mitigating Copy Bias in In-Context Learning through Neuron Pruningcited by 11
  11. 2025Calibration Across Layers: Understanding Calibration Evolution in LLMscited by 9
  12. 2025Geometry of Decision Making in Language Modelscited by 8
38 more
  1. 2026Closing the Confidence-Faithfulness Gap in Large Language Modelscited by 8
  2. 2025Transformers Don't Need LayerNorm at Inference Time: Scaling LayerNorm Removal to GPT-2 XL and the Implications for Mechanistic Interpretabilitycited by 7
  3. 2025Understanding and Mitigating Political Stance Cross-topic Generalization in Large Language Modelscited by 6
  4. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 5
  5. 2025Unsupervised Word-level Quality Estimation for Machine Translation Through the Lens of Annotators (Dis)agreementcited by 4
  6. 2025Isolating Culture Neurons in Multilingual Large Language Modelscited by 4
  7. 2026Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steeringcited by 4
  8. 2026The Confidence Manifold: Geometric Structure of Correctness Representations in Language Modelscited by 4
  9. 2026Wired for Overconfidence: A Mechanistic Perspective on Inflated Verbalized Confidence in LLMscited by 4
  10. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 4
  11. 2026The Weight Norm Sets the Grokking Timescale: A Causal Delay Lawcited by 4
  12. 2025Prediction hubs are context-informed frequent tokens in LLMscited by 3
  13. 2025Not a nuisance but a useful heuristic: Outlier dimensions favor frequent tokens in language modelscited by 3
  14. 2025Fine-Tuned LLMs Know They Don't Know: A Parameter-Efficient Approach to Recovering Honestycited by 3
  15. 2026Suppressing Final Layer Hidden State Jumps in Transformer Pretrainingcited by 3
  16. 2025Emergent Specialization: Rare Token Neurons in Language Modelscited by 2
  17. 2025Detecting and Pruning Prominent but Detrimental Neurons in Large Language Modelscited by 2
  18. 2025Understanding Post-Training Structural Changes in Large Language Modelscited by 2
  19. 2025Negative Pre-activations Differentiate Syntaxcited by 2
  20. 2026Correctness-Optimized Residual Activation Lens (CORAL): Transferrable and Calibration-Aware Inference-Time Steeringcited by 2
  21. 2025Promote, Suppress, Iterate: How Language Models Answer One-to-Many Factual Queriescited by 1
  22. 2025Understanding Gated Neurons in Transformers from Their Input-Output Functionalitycited by 1
  23. 2025Context Copying Modulation: The Role of Entropy Neurons in Managing Parametric and Contextual Knowledge Conflictscited by 1
  24. 2025From Shallow Humor to Metaphor: Towards Label-Free Harmful Meme Detection via LMM Agent Self-Improvementcited by 1
  25. 2026INTRYGUE: Induction-Aware Entropy Gating for Reliable RAG Uncertainty Estimationcited by 1
  26. 2026Towards Effective In-context Cross-domain Knowledge Transfer via Domain-invariant-neurons-based Retrievalcited by 1
  27. 2026Disentangling MLP Neuron Weights in Vocabulary Spacecited by 1
  28. 2026SIEVES: Selective Prediction Generalizes through Visual Evidence Scoringcited by 1
  29. 2026Massive Activations Are Architecturally Robust: A Controlled Scratch/Commitment Residual Stream Testcited by 1
  30. 2024Label Set Optimization via Activation Distribution Kurtosis for Zero-shot Classification with Generative Modelscited by 0
  31. 2025Distributed Specialization: Rare-Token Neurons in Large Language Modelscited by 0
  32. 2025Enhancing LLM Reasoning via Non-Human-Like Reasoning Path Preference Optimizationcited by 0
  33. 2025GEM: Generative Entropy-Guided Preference Modeling for Few-shot Alignment of LLMscited by 0
  34. 2025Sparse Neuron Ablation Triggers Catastrophic Collapse of the Language Core in Large Vision-Language Modelscited by 0
  35. 2026Entropy-Guided k-Guard Sampling for Long-Horizon Autoregressive Video Generationcited by 0
  36. 2026You Can Learn Tokenization End-to-End with Reinforcement Learningcited by 0
  37. 2026GLUScope: A Tool for Analyzing GLU Neurons in Transformer Language Modelscited by 0
  38. 2026On the Tip of the Tongue: Why LLMs Hallucinate Answers They Can Decodecited by 0

50 most cited of 57 All 57 (JSON)