mlsanityData to 14 Oct 2026

LFA applied to CNNs: Efficient Singular Value Decomposition of Convolutional Mappings by Local Fourier Analysis

arXiv 2506.05617 ↗First posted 5 Jun 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

The singular values of convolutional mappings encode interesting spectral properties, which can be used, e.g., to improve generalization and robustness of convolutional neural networks as well as to facilitate model compression. However, the computation of singular values is typically very resource-intensive. The naive approach involves unrolling the convolutional mapping along the input and channel dimensions into a large and sparse two-dimensional matrix, making the exact calculation of all singular values infeasible due to hardware limitations. In particular, this is true for matrices that represent convolutional mappings with large inputs and a high number of channels. Existing efficient methods leverage the Fast Fourier transformation (FFT) to transform convolutional mappings into the frequency domain, enabling the computation of singular values for matrices representing convolutions with larger input and channel dimensions. For a constant number of channels in a given convolution, an FFT can compute N singular values in O(N log N) complexity. In this work, we propose an approach of complexity O(N) based on local Fourier analysis, which additionally exploits the shift invariance of convolutional operators. We provide a theoretical analysis of our algorithm's runtime and validate its efficiency through numerical experiments. Our results demonstrate that our proposed method is scalable and offers a practical solution to calculate the entire set of singular values - along with the corresponding singular vectors if needed - for high-dimensional convolutional mappings.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 19 of its 28 references that are in the map. First posted on arXiv 5 Jun 2025.

Builds on

19 of 28 references.

References · 19

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
  3. 2017A PAC-Bayesian Approach to Spectrally-Normalized Margin Bounds for Neural Networkscited by 314
  4. 2018Regularisation of Neural Networks by Enforcing Lipschitz Continuitycited by 314
  5. 2014Exploiting Linear Structure Within Convolutional Networks for Efficient Evaluationcited by 306
  6. 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 293
  7. 2014Speeding up Convolutional Neural Networks with Low Rank Expansionscited by 284
  8. 2019Plug-and-Play Methods Provably Converge with Properly Trained Denoiserscited by 198
  9. 2018Lipschitz-Margin Training: Scalable Certification of Perturbation Invariance for Deep Neural Networkscited by 195
  10. 2022Language model compression with weighted low-rank factorizationcited by 195
  11. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 182
  12. 2015Accelerating Very Deep Convolutional Networks for Classification and Detectioncited by 109
7 more
  1. 2018The Singular Values of Convolutional Layerscited by 95
  2. 2018GroupReduce: Block-Wise Low-Rank Approximation for Neural Language Model Shrinkingcited by 32
  3. 2018Online Embedding Compression for Text Classification using Low Rank Matrix Factorizationcited by 22
  4. 2020MGIC: Multigrid-in-Channels Neural Network Architecturescited by 8
  5. 2022The SVD of Convolutional Weights: A CNN Interpretability Frameworkcited by 8
  6. 2024Invisible Backdoor Attack Through Singular Value Decompositioncited by 1
  7. 2025Poly-MgNet: Polynomial Building Blocks in Multigrid-Inspired ResNetscited by 1

Built on by

Not cited yet.