mlsanityData to 7 Oct 2026

Spectral Roll-off Points Variations: Exploring Useful Information in Feature Maps by Its Variations

arXiv 2102.00369 ↗First posted 31 Jan 2021cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Useful information (UI) is an elusive concept in neural networks. A quantitative measurement of UI is absent, despite the variations of UI can be recognized by prior knowledge. The communication bandwidth of feature maps decreases after downscaling operations, but UI flows smoothly after training due to lower Nyquist frequency. Inspired by the low-Nyqusit-frequency nature of UI, we propose the use of spectral roll-off points (SROPs) to estimate UI on variations. The computation of an SROP is extended from a 1-D signal to a 2-D image by the required rotation invariance in image classification tasks. SROP statistics across feature maps are implemented as layer-wise useful information estimates. We design sanity checks to explore SROP variations when UI variations are produced by variations in model input, model architecture and training stages. The variations of SROP is synchronizes with UI variations in various randomized and sufficiently trained model structures. Therefore, SROP variations is an accurate and convenient sign of UI variations, which promotes the explainability of data representations with respect to frequency-domain knowledge.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 26 of its 43 references that are in the map. First posted on arXiv 31 Jan 2021.

Builds on

26 of 43 references.

References · 26

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  6. 2017Understanding Black-box Predictions via Influence Functionscited by 1,714
  7. 2018On the Spectral Bias of Neural Networkscited by 1,321
  8. 2018Sanity Checks for Saliency Mapscited by 1,149
  9. 2017A Closer Look at Memorization in Deep Networkscited by 936
  10. 2019Fantastic Generalization Measures and Where to Find Themcited by 507
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  12. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
14 more
  1. 2019Frequency Principle: Fourier Analysis Sheds Light on Deep Neural Networkscited by 360
  2. 2019High Frequency Component Helps Explain the Generalization of Convolutional Neural Networkscited by 310
  3. 2019Intrinsic dimension of data representations in deep neural networkscited by 276
  4. 2018Estimating the intrinsic dimension of datasets by a minimal neighborhood informationcited by 246
  5. 2014Understanding image representations by measuring their equivariance and equivalencecited by 240
  6. 2020Learning in the Frequency Domaincited by 196
  7. 2018Training behavior of deep neural network in frequency domaincited by 174
  8. 2020A Theory of Usable Information Under Computational Constraintscited by 153
  9. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  10. 2016Exploiting Cyclic Symmetry in Convolutional Neural Networkscited by 91
  11. 2020Explaining Knowledge Distillation by Quantifying the Knowledgecited by 65
  12. 2020Spatially Adaptive Inference with Stochastic Feature Sampling and Interpolationcited by 51
  13. 2019Towards Task and Architecture-Independent Generalization Gap Predictorscited by 18
  14. 2019Knowledge Consistency between Neural Networks and Beyondcited by 16

Built on by

Not cited yet.