mlsanityData to 14 Oct 2026

Sum-of-Parts: Self-Attributing Neural Networks with End-to-End Learning of Feature Groups

arXiv 2310.16316 ↗First posted 25 Oct 2023cs.LG cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Self-attributing neural networks (SANNs) present a potential path towards interpretable models for high-dimensional problems, but often face significant trade-offs in performance. In this work, we formally prove a lower bound on errors of per-feature SANNs, whereas group-based SANNs can achieve zero error and thus high performance. Motivated by these insights, we propose Sum-of-Parts (SOP), a framework that transforms any differentiable model into a group-based SANN, where feature groups are learned end-to-end without group supervision. SOP achieves state-of-the-art performance for SANNs on vision and language tasks, and we validate that the groups are interpretable on a range of quantitative and semantic metrics. We further validate the utility of SOP explanations in model debugging and cosmological scientific discovery. Our code is available at https://github.com/BrachioLab/sop

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 44 of its 81 references that are in the map. First posted on arXiv 25 Oct 2023.

Builds on

44 of 81 references.

References · 44

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  5. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  6. 2017A Unified Approach to Interpreting Model Predictionscited by 6,555
  7. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,070
  8. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  9. 2018Sanity Checks for Saliency Mapscited by 1,156
  10. 2020Concept Bottleneck Modelscited by 1,032
  11. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  12. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
32 more
  1. 2019ERASER: A Benchmark to Evaluate Rationalized NLP Modelscited by 530
  2. 2018Towards Robust Interpretability with Self-Explaining Neural Networkscited by 505
  3. 2015Evaluating the visualization of what a Deep Neural Network has learnedcited by 419
  4. 2019A Survey on Explainable Artificial Intelligence (XAI): Towards Medical XAIcited by 404
  5. 2016From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classificationcited by 341
  6. 2020Neural Additive Models: Interpretable Machine Learning with Neural Netscited by 319
  7. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  8. 2022From Anecdotal Evidence to Quantitative Evaluation Methods: A Systematic Review on Evaluating Explainable AIcited by 302
  9. 2023Faithful Chain-of-Thought Reasoningcited by 288
  10. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  11. 2018Manipulating and Measuring Model Interpretabilitycited by 256
  12. 2019Full-Gradient Representation for Neural Network Visualizationcited by 191
  13. 2020Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior?cited by 190
  14. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  15. 2020Learning to Faithfully Rationalize by Constructioncited by 113
  16. 2021HIVE: Evaluating the Human Interpretability of Visual Explanationscited by 105
  17. 2021Large-scale Unsupervised Semantic Segmentationcited by 101
  18. 2022B-cos Networks: Alignment is All We Need for Interpretabilitycited by 88
  19. 2022Impossibility Theorems for Feature Attributioncited by 78
  20. 2020Interpretable and Accurate Fine-grained Recognition via Region Groupingcited by 75
  21. 2020How does this interaction affect me? Interpretable attribution for feature interactionscited by 73
  22. 2020Fairwashing Explanations with Off-Manifold Detergentcited by 72
  23. 2020SCOUT: Self-aware Discriminant Counterfactual Explanationscited by 52
  24. 2019Relative Attributing Propagation: Interpreting the Comparative Contributions of Individual Units in Deep Neural Networkscited by 49
  25. 2022Learning to Estimate Shapley Values with Vision Transformerscited by 45
  26. 2022Towards Formal XAI: Formally Approximate Minimal Explanations of Neural Networkscited by 42
  27. 2022Missingness Bias in Model Debuggingcited by 38
  28. 2021Fast Axiomatic Attribution for Neural Networkscited by 35
  29. 2018Towards Explanation of DNN-based Prediction with Guided Feature Inversioncited by 31
  30. 2023Stability Guarantees for Feature Attributions with Multiplicative Smoothingcited by 14
  31. 2021From Kepler to Newton: Explainable AI for Sciencecited by 13
  32. 2019Scalable Explanation of Inferences on Large Graphscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 2.

6 papers.

  1. 2026Interpretability Can Be Actionablecited by 7
  2. 2024DISCRET: Synthesizing Faithful Explanations For Treatment Effect Estimationcited by 2
  3. 2025T-FIX: Text-Based Explanations with Features Interpretable to eXpertscited by 2
  4. 2026Provably Explaining Neural Additive Modelscited by 1
  5. 2025When LRP Diverges from Leave-One-Out in Transformerscited by 0
  6. 2025TRANSPORTER: Transferring Visual Semantics from VLM Manifoldscited by 0