# Revisiting Sparse Convolutional Model for Visual Recognition

arXiv: https://arxiv.org/abs/2210.12945 · HTML: https://mlsanity.com/paper/2210.12945/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 60 references that are in the map.
- First posted on arXiv 24 Oct 2022.

## Builds on (most cited first, 30 of 35)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Benchmarking Neural Network Robustness to Common Corruptions and Perturbations](https://mlsanity.com/paper/1903.12261/) (2019)
- [Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Maps](https://mlsanity.com/paper/1312.6034/) (2013)
- [Visualizing and Understanding Convolutional Networks](https://mlsanity.com/paper/1311.2901/) (2013)
- [The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization](https://mlsanity.com/paper/2006.16241/) (2020)
- [Understanding deep learning requires rethinking generalization](https://mlsanity.com/paper/1611.03530/) (2016)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples](https://mlsanity.com/paper/1802.00420/) (2018)
- [Adversarial Machine Learning at Scale](https://mlsanity.com/paper/1611.01236/) (2016)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [Deep Equilibrium Models](https://mlsanity.com/paper/1909.01377/) (2019)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Understanding Deep Image Representations by Inverting Them](https://mlsanity.com/paper/1412.0035/) (2014)
- [A Fourier Perspective on Model Robustness in Computer Vision](https://mlsanity.com/paper/1906.08988/) (2019)
- [Inverting Visual Representations with Convolutional Networks](https://mlsanity.com/paper/1506.02753/) (2015)
- [Improving the Robustness of Deep Neural Networks via Stability Training](https://mlsanity.com/paper/1604.04326/) (2016)
- [Multiscale Deep Equilibrium Models](https://mlsanity.com/paper/2006.08656/) (2020)
- [Implicit Deep Learning](https://mlsanity.com/paper/1908.06315/) (2019)
- [Convolutional Neural Networks Analyzed via Convolutional Sparse Coding](https://mlsanity.com/paper/1607.08194/) (2016)
- [Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picture](https://mlsanity.com/paper/1511.03607/) (2015)
- [Sparse Modeling for Image and Vision Processing](https://mlsanity.com/paper/1411.3230/) (2014)
- [Deep Isometric Learning for Visual Recognition](https://mlsanity.com/paper/2006.16992/) (2020)
- [Learned Convolutional Sparse Coding](https://mlsanity.com/paper/1711.00328/) (2017)
- [Deep K-SVD Denoising](https://mlsanity.com/paper/1909.13164/) (2019)
- [Adversarial Robustness of Supervised Sparse Coding](https://mlsanity.com/paper/2010.12088/) (2020)
- [Architectural Adversarial Robustness: The Case for Deep Pursuit](https://mlsanity.com/paper/2011.14427/) (2020)
- [Structured Local Optima in Sparse Blind Deconvolution](https://mlsanity.com/paper/1806.00338/) (2018)
- [Implicit Feature Pyramid Network for Object Detection](https://mlsanity.com/paper/2012.13563/) (2020)
- [Convolutional Normalization: Improving Deep Convolutional Network Robustness and Training](https://mlsanity.com/paper/2103.00673/) (2021)

## Built on by (most cited first, 14 of 14)

- [Scaling White-Box Transformers for Vision](https://mlsanity.com/paper/2405.20299/) (2024)
- [LCANets++: Robust Audio Classification using Multi-layer Neural Networks with Lateral Competition](https://mlsanity.com/paper/2308.12882/) (2023)
- [SC-MIL: Sparsely Coded Multiple Instance Learning for Whole Slide Image Classification](https://mlsanity.com/paper/2311.00048/) (2023)
- [Closed-Loop Transcription via Convolutional Sparse Coding](https://mlsanity.com/paper/2302.09347/) (2023)
- [Which2comm: An Efficient Collaborative Perception Framework for 3D Object Detection](https://mlsanity.com/paper/2503.17175/) (2025)
- [SINET: Sparsity-driven Interpretable Neural Network for Underwater Image Enhancement](https://mlsanity.com/paper/2409.01022/) (2024)
- [Boosting Adversarial Robustness and Generalization with Dictionary Structure](https://mlsanity.com/paper/2502.00834/) (2025)
- [An NMF-Based Building Block for Interpretable Neural Networks With Continual Learning](https://mlsanity.com/paper/2311.11485/) (2023)
- [A Masked Pruning Approach for Dimensionality Reduction in Communication-Efficient Federated Learning Systems](https://mlsanity.com/paper/2312.03889/) (2023)
- [IB-AdCSCNet:Adaptive Convolutional Sparse Coding Network Driven by Information Bottleneck](https://mlsanity.com/paper/2405.14192/) (2024)
- [Omni-Dimensional Frequency Learner for General Time Series Analysis](https://mlsanity.com/paper/2407.10419/) (2024)
- [Hyperspectral Unmixing with 3D Convolutional Sparse Coding and Projected Simplex Volume Maximization](https://mlsanity.com/paper/2512.05674/) (2025)
- [Training-Adaptive Convolutional Sparse Coding via Information Bottleneck for Robust Visual Representation](https://mlsanity.com/paper/2609.19122/) (2026)
- [ConEx: Human-Interpretable Saliency Maps via Concept-Aware Attribution](https://mlsanity.com/paper/2610.04605/) (2026)

## Abstract (arXiv)

Despite strong empirical performance for image classification, deep neural networks are often regarded as ``black boxes'' and they are difficult to interpret. On the other hand, sparse convolutional models, which assume that a signal can be expressed by a linear combination of a few elements from a convolutional dictionary, are powerful tools for analyzing natural images with good theoretical interpretability and biological plausibility. However, such principled models have not demonstrated competitive performance when compared with empirically designed deep networks. This paper revisits the sparse convolutional modeling for image classification and bridges the gap between good empirical performance (of deep learning) and good interpretability (of sparse convolutional models). Our method uses differentiable optimization layers that are defined from convolutional sparse coding as drop-in replacements of standard convolutional layers in conventional deep neural networks. We show that such models have equally strong empirical performance on CIFAR-10, CIFAR-100, and ImageNet datasets when compared to conventional neural networks. By leveraging stable recovery property of sparse modeling, we further show that such models can be much more robust to input corruptions as well as adversarial perturbations in testing through a simple proper trade-off between sparse regularization and data reconstruction terms. Source code can be found at https://github.com/Delay-Xili/SDNet.