# Leveraging Sparse Linear Layers for Debuggable Deep Networks

arXiv: https://arxiv.org/abs/2105.04857 · HTML: https://mlsanity.com/paper/2105.04857/

## Key facts

- Cited by 82 papers in the map (data through 14 Oct 2026).
- Builds on 40 of its 64 references that are in the map.
- First posted on arXiv 11 May 2021.

## Builds on (most cited first, 30 of 40)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Towards Deep Learning Models Resistant to Adversarial Attacks](https://mlsanity.com/paper/1706.06083/) (2017)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning both Weights and Connections for Efficient Neural Networks](https://mlsanity.com/paper/1506.02626/) (2015)
- [Understanding Black-box Predictions via Influence Functions](https://mlsanity.com/paper/1703.04730/) (2017)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Sanity Checks for Saliency Maps](https://mlsanity.com/paper/1810.03292/) (2018)
- [Pruning Filters for Efficient ConvNets](https://mlsanity.com/paper/1608.08710/) (2016)
- [SmoothGrad: removing noise by adding noise](https://mlsanity.com/paper/1706.03825/) (2017)
- [Adversarial Examples Are Not Bugs, They Are Features](https://mlsanity.com/paper/1905.02175/) (2019)
- [Robustness May Be at Odds with Accuracy](https://mlsanity.com/paper/1805.12152/) (2018)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Recognition in Terra Incognita](https://mlsanity.com/paper/1807.04975/) (2018)
- [What is the State of Neural Network Pruning?](https://mlsanity.com/paper/2003.03033/) (2020)
- [Interpretation of Neural Networks is Fragile](https://mlsanity.com/paper/1710.10547/) (2017)
- [Fooling LIME and SHAP: Adversarial Attacks on Post hoc Explanation Methods](https://mlsanity.com/paper/1911.02508/) (2019)
- [Nuanced Metrics for Measuring Unintended Bias with Real Data for Text Classification](https://mlsanity.com/paper/1903.04561/) (2019)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Noise or Signal: The Role of Image Backgrounds in Object Recognition](https://mlsanity.com/paper/2006.09994/) (2020)
- [Understanding Neural Networks Through Deep Visualization](https://mlsanity.com/paper/1506.06579/) (2015)
- [Actionable Recourse in Linear Classification](https://mlsanity.com/paper/1809.06514/) (2018)
- [Do Adversarially Robust ImageNet Models Transfer Better?](https://mlsanity.com/paper/2007.08489/) (2020)
- [Counterfactual Visual Explanations](https://mlsanity.com/paper/1904.07451/) (2019)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [On Completeness-aware Concept-Based Explanations in Deep Neural Networks](https://mlsanity.com/paper/1910.07969/) (2019)

## Built on by (most cited first, 30 of 82)

- [SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation](https://mlsanity.com/paper/2310.12508/) (2023)
- [Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts](https://mlsanity.com/paper/2406.12845/) (2024)
- [Label-Free Concept Bottleneck Models](https://mlsanity.com/paper/2304.06129/) (2023)
- [Model Sparsity Can Simplify Machine Unlearning](https://mlsanity.com/paper/2304.04934/) (2023)
- [Datamodels: Predicting Predictions from Training Data](https://mlsanity.com/paper/2202.00622/) (2022)
- [DICE: Leveraging Sparsification for Out-of-Distribution Detection](https://mlsanity.com/paper/2111.09805/) (2021)
- [Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networks](https://mlsanity.com/paper/2207.13243/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Salient ImageNet: How to discover spurious features in Deep Learning?](https://mlsanity.com/paper/2110.04301/) (2021)
- [Distilling Model Failures as Directions in Latent Space](https://mlsanity.com/paper/2206.14754/) (2022)
- [Editing a classifier by rewriting its prediction rules](https://mlsanity.com/paper/2112.01008/) (2021)
- [VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidance](https://mlsanity.com/paper/2408.01432/) (2024)
- [A Comprehensive Study of Image Classification Model Sensitivity to Foregrounds, Backgrounds, and Visual Attributes](https://mlsanity.com/paper/2201.10766/) (2022)
- [Explainable and Interpretable Multimodal Large Language Models: A Comprehensive Survey](https://mlsanity.com/paper/2412.02104/) (2024)
- [FFCV: Accelerating Training by Removing Data Bottlenecks](https://mlsanity.com/paper/2306.12517/) (2023)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Images](https://mlsanity.com/paper/2305.19164/) (2023)
- [Weight-sparse transformers have interpretable circuits](https://mlsanity.com/paper/2511.13653/) (2025)
- [The Road to Explainability is Paved with Bias: Measuring the Fairness of Explanations](https://mlsanity.com/paper/2205.03295/) (2022)
- [Spurious Features Everywhere -- Large-Scale Detection of Harmful Spurious Features in ImageNet](https://mlsanity.com/paper/2212.04871/) (2022)
- [Missingness Bias in Model Debugging](https://mlsanity.com/paper/2204.08945/) (2022)
- [MultiViz: Towards Visualizing and Understanding Multimodal Models](https://mlsanity.com/paper/2207.00056/) (2022)
- [DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanations](https://mlsanity.com/paper/2203.02013/) (2022)
- [3DB: A Framework for Debugging Computer Vision Models](https://mlsanity.com/paper/2106.03805/) (2021)
- [Coarse-to-Fine Concept Bottleneck Models](https://mlsanity.com/paper/2310.02116/) (2023)
- [ModelDiff: A Framework for Comparing Learning Algorithms](https://mlsanity.com/paper/2211.12491/) (2022)
- [How Well do Feature Visualizations Support Causal Understanding of CNN Activations?](https://mlsanity.com/paper/2106.12447/) (2021)
- [Language Guided Concept Bottleneck Models for Interpretable Continual Learning](https://mlsanity.com/paper/2503.23283/) (2025)
- [FACTS: First Amplify Correlations and Then Slice to Discover Bias](https://mlsanity.com/paper/2309.17430/) (2023)
- [Sparse Linear Concept Discovery Models](https://mlsanity.com/paper/2308.10782/) (2023)

## Abstract (arXiv)

We show how fitting sparse linear models over learned deep feature representations can lead to more debuggable neural networks. These networks remain highly accurate while also being more amenable to human interpretation, as we demonstrate quantiatively via numerical and human experiments. We further illustrate how the resulting sparse explanations can help to identify spurious correlations, explain misclassifications, and diagnose model biases in vision and language tasks. The code for our toolkit can be found at https://github.com/madrylab/debuggabledeepnetworks.