mlsanityData to 14 Oct 2026

Data-induced multiscale losses and efficient multirate gradient descent schemes

arXiv 2402.03021 ↗First posted 5 Feb 2024cs.LG cs.NA math.NAcited by 0

Not on the map. Why?

Abstract (arXiv)

This paper investigates the impact of multiscale data on machine learning algorithms, particularly in the context of deep learning. A dataset is multiscale if its distribution shows large variations in scale across different directions. This paper reveals multiscale structures in the loss landscape, including its gradients and Hessians inherited from the data. Correspondingly, it introduces a novel gradient descent approach, drawing inspiration from multiscale algorithms used in scientific computing. This approach seeks to transcend empirical learning rate selection, offering a more systematic, data-informed strategy to enhance training efficiency, especially in the later stages.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 59 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

23 of 59 references.

References · 23

  1. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  2. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  4. 2019On the Variance of the Adaptive Learning Rate and Beyondcited by 772
  5. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 659
  6. 2021The Intrinsic Dimension of Images and Its Impact on Learningcited by 315
  7. 2019Intrinsic dimension of data representations in deep neural networkscited by 282
  8. 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
  9. 2016The Landscape of Empirical Risk for Non-convex Lossescited by 86
  10. 2022Verifying the Union of Manifolds Hypothesis for Image Datacited by 64
  11. 2021Large Learning Rate Tames Homogeneity: Convergence and Balancing Effectcited by 50
  12. 2022Beyond the Quadratic Approximation: the Multiscale Structure of Neural Network Loss Landscapescited by 43
11 more
  1. 2020Stochasticity of Deterministic Gradient Descent: Large Learning Rate for Multiscale Objective Functioncited by 26
  2. 2023Provably Faster Gradient Descent via Long Stepscited by 20
  3. 2022Branch-and-Bound Performance Estimation Programming: A Unified Methodology for Constructing Optimal Optimization Methodscited by 16
  4. 2022Effects of Data Geometry in Early Deep Learningcited by 12
  5. 2021Provable Super-Convergence with a Large Cyclical Learning Ratecited by 10
  6. 2021On the Convergence of Stochastic Gradient Descent with Bandwidth-based Step Sizecited by 9
  7. 2022Semi-Supervised Manifold Learning with Complexity Decoupled Chart Autoencoderscited by 8
  8. 2021Big-Step-Little-Step: Efficient Gradient Methods for Objectives with Multiple Scalescited by 7
  9. 2022Side Effects of Learning from Low-dimensional Data Embedded in a Euclidean Spacecited by 6
  10. 2018Heuristic Framework for Multi-Scale Testing of the Multi-Manifold Hypothesiscited by 5
  11. 2023Linear Regression on Manifold Structured Data: the Impact of Extrinsic Geometry on Solutionscited by 3

Built on by

Not cited yet.