mlsanityData to 14 Oct 2026

RACE: Scalable Statistical Estimation of Functional Consistency in LLM Neurons

arXiv 2608.24758 ↗First posted 25 Aug 2026cs.AIcited by 0

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Discovering stable neuron behavior across entire domains remains a challenge in mechanistic interpretability. Existing methods often rely on instance-level point estimates or computationally expensive procedures, which either obscure population-level variability or limit scalable domain-wide analysis. We present RACE (Residual Alignment for Consistency Estimation), a forward-pass statistical framework that evaluates the domain-wide functional consistency of Transformer neurons. Compared with gradient-based point estimates, RACE produces neuron rankings that yield more domain-specific effects under perturbation. Token-distribution shifts support the connection between the selected neurons and the target domain, while scoring requires roughly one-hundredth of the computational overhead of the gradient-based methods. Code is available at https://github.com/Nexround/RACE.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 35 of its 52 references that are in the map. First posted on arXiv 25 Aug 2026.

Builds on

35 of 52 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Discovering Transformer Circuits via a Hybrid Attribution and Pruning Framework28 Sep
  2. 2024Hypothesis Testing the Circuit Hypothesis in LLMs16 Oct
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretability28 Apr

Other references · 32

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  3. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  6. 2016Pointer Sentinel Mixture Modelscited by 3,096
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2017Learning Important Features Through Propagating Activation Differencescited by 1,458
  10. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
20 more
  1. 2024The FineWeb Datasets: Decanting the Web for the Finest Text Data at Scalecited by 1,160
  2. 2018Sanity Checks for Saliency Mapscited by 1,156
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,121
  4. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
  5. 2019Are Sixteen Heads Really Better than One?cited by 816
  6. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 623
  7. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  8. 2020Captum: A unified and generic model interpretability library for PyTorchcited by 547
  9. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  10. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 528
  11. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 235
  12. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 156
  13. 2024AttnLRP: Attention-Aware Layer-Wise Relevance Propagation for Transformerscited by 120
  14. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 115
  15. 2023Neurons in Large Language Models: Dead, N-gram, Positionalcited by 88
  16. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 76
  17. 2023Journey to the Center of the Knowledge Neurons: Discoveries of Language-Independent Knowledge Neurons and Degenerate Knowledge Neuronscited by 56
  18. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  19. 2024Towards Understanding Multi-Task Learning (Generalization) of LLMs via Detecting and Exploring Task-Specific Neuronscited by 25
  20. 2023An Adversarial Example for Direct Logit Attribution: Memory Management in GELU-4Lcited by 11

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Disentangling Polysemantic Neurons with a Null-Calibrated Polysemanticity Index and Causal Patch Interventions
  2. 2025Do LLMs Signal When They're Right? Evidence from Neuron Agreement
  3. 2025Faithful and Stable Neuron Explanations for Trustworthy Mechanistic Interpretability
  4. 2026A Few Bad Neurons: Isolating and Surgically Correcting Sycophancy
  5. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
  6. 2026Quantifying LLM Attention-Head Stability: Implications for Circuit Universality
  7. 2025Task-Specific Data Selection for Instruction Tuning via Monosemantic Neuronal Activations
  8. 2026Targeted Neuron Modulation via Contrastive Pair Search