# On Multi-Layer Basis Pursuit, Efficient Algorithms and Convolutional Neural Networks

arXiv: https://arxiv.org/abs/1806.00701 · HTML: https://mlsanity.com/paper/1806.00701/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Jun 2018.

## Built on by (most cited first, 21 of 21)

- [Learning to Optimize: A Primer and A Benchmark](https://mlsanity.com/paper/2103.12828/) (2021)
- [Can stable and accurate neural networks be computed? -- On the barriers of deep learning and Smale's 18th problem](https://mlsanity.com/paper/2101.08286/) (2021)
- [What's in a Prior? Learned Proximal Networks for Inverse Problems](https://mlsanity.com/paper/2310.14344/) (2023)
- [Adversarial robustness of sparse local Lipschitz predictors](https://mlsanity.com/paper/2202.13216/) (2022)
- [Phase Collapse in Neural Networks](https://mlsanity.com/paper/2110.05283/) (2021)
- [CSC-Unet: A Novel Convolutional Sparse Coding Strategy Based Neural Network for Semantic Segmentation](https://mlsanity.com/paper/2108.00408/) (2021)
- [Designing Rotationally Invariant Neural Networks from PDEs and Variational Methods](https://mlsanity.com/paper/2108.13993/) (2021)
- [Medical Image Segmentation via Sparse Coding Decoder](https://mlsanity.com/paper/2310.10957/) (2023)
- [Convergence Analysis for Deep Sparse Coding via Convolutional Neural Networks](https://mlsanity.com/paper/2408.05540/) (2024)
- [Reframing Neural Networks: Deep Structure in Overcomplete Representations](https://mlsanity.com/paper/2103.05804/) (2021)
- [Deep learning based dictionary learning and tomographic image reconstruction](https://mlsanity.com/paper/2108.11730/) (2021)
- [Pivotal Auto-Encoder via Self-Normalizing ReLU](https://mlsanity.com/paper/2406.16052/) (2024)
- [Towards Explainable Multi-Party Learning: A Contrastive Knowledge Sharing Framework](https://mlsanity.com/paper/2104.06670/) (2021)
- [Learned Interpretable Residual Extragradient ISTA for Sparse Coding](https://mlsanity.com/paper/2106.11970/) (2021)
- [A Sparse Coding Interpretation of Neural Networks and Theoretical Implications](https://mlsanity.com/paper/2108.06622/) (2021)
- [WSEBP: A Novel Width-depth Synchronous Extension-based Basis Pursuit Algorithm for Multi-Layer Convolutional Sparse Coding](https://mlsanity.com/paper/2203.14856/) (2022)
- [Learning unfolded networks with a cyclic group structure](https://mlsanity.com/paper/2211.09238/) (2022)
- [CQnet: convex-geometric interpretation and constraining neural-network trajectories](https://mlsanity.com/paper/2302.10895/) (2023)
- [Learning Linear Groups in Neural Networks](https://mlsanity.com/paper/2305.18552/) (2023)
- [A statistical perspective on algorithm unrolling models for inverse problems](https://mlsanity.com/paper/2311.06395/) (2023)
- [Specific Algorithmic Interpretability of Neural Networks: A Case Study on Textures](https://mlsanity.com/paper/2610.04413/) (2026)

## Abstract (arXiv)

Parsimonious representations are ubiquitous in modeling and processing information. Motivated by the recent Multi-Layer Convolutional Sparse Coding (ML-CSC) model, we herein generalize the traditional Basis Pursuit problem to a multi-layer setting, introducing similar sparse enforcing penalties at different representation layers in a symbiotic relation between synthesis and analysis sparse priors. We explore different iterative methods to solve this new problem in practice, and we propose a new Multi-Layer Iterative Soft Thresholding Algorithm (ML-ISTA), as well as a fast version (ML-FISTA). We show that these nested first order algorithms converge, in the sense that the function value of near-fixed points can get arbitrarily close to the solution of the original problem. We further show how these algorithms effectively implement particular recurrent convolutional neural networks (CNNs) that generalize feed-forward ones without introducing any parameters. We present and analyze different architectures resulting unfolding the iterations of the proposed pursuit algorithms, including a new Learned ML-ISTA, providing a principled way to construct deep recurrent CNNs. Unlike other similar constructions, these architectures unfold a global pursuit holistically for the entire network. We demonstrate the emerging constructions in a supervised learning setting, consistently improving the performance of classical CNNs while maintaining the number of parameters constant.