mlsanityData to 7 Oct 2026

Uncovering Model Processing Strategies with Non-Negative Per-Example Fisher Factorization

arXiv 2310.04649 ↗First posted 7 Oct 2023cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

We introduce NPEFF (Non-Negative Per-Example Fisher Factorization), an interpretability method that aims to uncover strategies used by a model to generate its predictions. NPEFF decomposes per-example Fisher matrices using a novel decomposition algorithm that learns a set of components represented by learned rank-1 positive semi-definite matrices. Through a combination of human evaluation and automated analysis, we demonstrate that these NPEFF components correspond to model processing strategies for a variety of language models and text processing tasks. We further show how to construct parameter perturbations from NPEFF components to selectively disrupt a given component's role in the model's processing. Along with conducting extensive ablation studies, we include experiments to show how NPEFF can be used to analyze and mitigate collateral effects of unlearning and use NPEFF to study in-context learning. Furthermore, we demonstrate the advantages of NPEFF over baselines such as gradient clustering and using sparse autoencoders for dictionary learning over model activations. We release the code used in this work.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 52 of its 153 references that are in the map. First posted on arXiv 7 Oct 2023.

Builds on

52 of 153 references.

References · 52

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  4. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  8. 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,348
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  12. 2019Root Mean Square Layer Normalizationcited by 1,425
38 more
  1. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  2. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2021What Makes Good In-Context Examples for GPT-3?cited by 1,184
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  6. 2017SmoothGrad: removing noise by adding noisecited by 1,097
  7. 2022Mass-Editing Memory in a Transformercited by 976
  8. 2022In-context Learning and Induction Headscited by 930
  9. 2023Progress measures for grokking via mechanistic interpretabilitycited by 822
  10. 2015Optimizing Neural Networks with Kronecker-factored Approximate Curvaturecited by 776
  11. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  12. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  14. 2014New insights and perspectives on the natural gradient methodcited by 435
  15. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  16. 2023Studying Large Language Model Generalization with Influence Functionscited by 311
  17. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  18. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  19. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  20. 2019Task2Vec: Task Embedding for Meta-Learningcited by 197
  21. 2022Active Example Selection for In-Context Learningcited by 195
  22. 2022Language model compression with weighted low-rank factorizationcited by 192
  23. 2023PMET: Precise Model Editing in a Transformercited by 170
  24. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  25. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  26. 2022Impact of Pretraining Term Frequencies on Few-Shot Reasoningcited by 142
  27. 2022If Influence Functions are the Answer, Then What is the Question?cited by 142
  28. 2016A Kronecker-factored approximate Fisher matrix for convolution layerscited by 138
  29. 2023The Quantization Model of Neural Scalingcited by 137
  30. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  31. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  32. 2017Critical Learning Periods in Deep Neural Networkscited by 78
  33. 2019On the interplay between noise and curvature and its effect on optimization and generalizationcited by 44
  34. 2021Measuring Data Leakage in Machine-Learning Models with Fisher Informationcited by 31
  35. 2022PipeFisher: Efficient Training of Large Language Models Using Pipelining and Fisher Information Matricescited by 24
  36. 2021On the Variance of the Fisher Information for Deep Learningcited by 22
  37. 2020Scalable and Practical Natural Gradient for Large-Scale Deep Learningcited by 15
  38. 2022Efficient Approximations of the Fisher Matrix in Neural Networks using Kronecker Product Singular Value Decompositioncited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Stochastic Parameter Decompositioncited by 14
  2. 2025Identifying Sparsely Active Circuits Through Local Loss Landscape Decompositioncited by 5