# Multi-Layer Convolutional Sparse Modeling: Pursuit and Dictionary Learning

arXiv: https://arxiv.org/abs/1708.08705 · HTML: https://mlsanity.com/paper/1708.08705/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2017.

## Built on by (most cited first, 14 of 14)

- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [Neural Response Interpretation through the Lens of Critical Pathways](https://mlsanity.com/paper/2103.16886/) (2021)
- [Revisiting Sparse Convolutional Model for Visual Recognition](https://mlsanity.com/paper/2210.12945/) (2022)
- [Phase Collapse in Neural Networks](https://mlsanity.com/paper/2110.05283/) (2021)
- [Sparsity-Probe: Analysis tool for Deep Learning Models](https://mlsanity.com/paper/2105.06849/) (2021)
- [Faster Convergence in Deep-Predictive-Coding Networks to Learn Deeper Representations](https://mlsanity.com/paper/2101.06848/) (2021)
- [Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks](https://mlsanity.com/paper/2408.05540/) (2024)
- [Deep learning based dictionary learning and tomographic image reconstruction](https://mlsanity.com/paper/2108.11730/) (2021)
- [Pivotal Auto-Encoder via Self-Normalizing ReLU](https://mlsanity.com/paper/2406.16052/) (2024)
- [CS-VLM: Compressed Sensing Attention for Efficient Vision-Language Representation Learning](https://mlsanity.com/paper/2507.02957/) (2025)
- [Improved Image Generation via Sparse Modeling](https://mlsanity.com/paper/2104.00464/) (2021)
- [Learned Interpretable Residual Extragradient ISTA for Sparse Coding](https://mlsanity.com/paper/2106.11970/) (2021)
- [A Sparse Coding Interpretation of Neural Networks and Theoretical Implications](https://mlsanity.com/paper/2108.06622/) (2021)
- [WSEBP: A Novel Width-depth Synchronous Extension-based Basis Pursuit Algorithm for Multi-Layer Convolutional Sparse Coding](https://mlsanity.com/paper/2203.14856/) (2022)

## Abstract (arXiv)

The recently proposed Multi-Layer Convolutional Sparse Coding (ML-CSC) model, consisting of a cascade of convolutional sparse layers, provides a new interpretation of Convolutional Neural Networks (CNNs). Under this framework, the computation of the forward pass in a CNN is equivalent to a pursuit algorithm aiming to estimate the nested sparse representation vectors -- or feature maps -- from a given input signal. Despite having served as a pivotal connection between CNNs and sparse modeling, a deeper understanding of the ML-CSC is still lacking: there are no pursuit algorithms that can serve this model exactly, nor are there conditions to guarantee a non-empty model. While one can easily obtain signals that approximately satisfy the ML-CSC constraints, it remains unclear how to simply sample from the model and, more importantly, how one can train the convolutional filters from real data. In this work, we propose a sound pursuit algorithm for the ML-CSC model by adopting a projection approach. We provide new and improved bounds on the stability of the solution of such pursuit and we analyze different practical alternatives to implement this in practice. We show that the training of the filters is essential to allow for non-trivial signals in the model, and we derive an online algorithm to learn the dictionaries from real data, effectively resulting in cascaded sparse convolutional layers. Last, but not least, we demonstrate the applicability of the ML-CSC model for several applications in an unsupervised setting, providing competitive results. Our work represents a bridge between matrix factorization, sparse dictionary learning and sparse auto-encoders, and we analyze these connections in detail.