Exploring and Exploiting the Asymmetric Valley of Deep Neural Networks
Not on the map. Why?
Abstract (arXiv)
Exploring the loss landscape offers insights into the inherent principles of deep neural networks (DNNs). Recent work suggests an additional asymmetry of the valley beyond the flat and sharp ones, yet without thoroughly examining its causes or implications. Our study methodically explores the factors affecting the symmetry of DNN valleys, encompassing (1) the dataset, network architecture, initialization, and hyperparameters that influence the convergence point; and (2) the magnitude and direction of the noise for 1D visualization. Our major observation shows that the it degree of sign consistency between the noise and the convergence point is a critical indicator of valley symmetry. Theoretical insights from the aspects of ReLU activation and softmax function could explain the interesting phenomenon. Our discovery propels novel understanding and applications in the scenario of Model Fusion: (1) the efficacy of interpolating separate models significantly correlates with their sign consistency ratio, and (2) imposing sign alignment during federated learning emerges as an innovative approach for model parameter alignment.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 60 of its 77 references that are in the map. First posted on arXiv 21 May 2024.
Builds on
60 of 77 references.
References · 60
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
- 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2018Federated Optimization in Heterogeneous Networkscited by 2,971
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
- 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
38 more
- 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
- 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
- 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
- 2018Federated Learning with Non-IID Datacited by 1,099
- 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
- 2021Model-Contrastive Federated Learningcited by 762
- 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
- 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
- 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
- 2017Exploring Generalization in Deep Learningcited by 595
- 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
- 2020Federated Learning with Matched Averagingcited by 564
- 2021Federated Learning Based on Dynamic Regularizationcited by 555
- 2019Fantastic Generalization Measures and Where to Find Themcited by 507
- 2017Sharp Minima Can Generalize For Deep Netscited by 462
- 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
- 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
- 2014On the Number of Linear Regions of Deep Neural Networkscited by 378
- 2020What is being transferred in transfer learning?cited by 365
- 2016Entropy-SGD: Biasing Gradient Descent Into Wide Valleyscited by 357
- 2018How Does Batch Normalization Help Optimization?cited by 355
- 2017Snapshot Ensembles: Train 1, get M for freecited by 349
- 2021Gradient Projection Memory for Continual Learningcited by 338
- 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
- 2019The Non-IID Data Quagmire of Decentralized Machine Learningcited by 303
- 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
- 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
- 2019Model Fusion via Optimal Transportcited by 217
- 2014Qualitatively characterizing neural network optimization problemscited by 186
- 2020The Break-Even Point on Optimization Trajectories of Deep Neural Networkscited by 149
- 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
- 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
- 2019Asymmetric Valleys: Beyond Sharp and Flat Local Minimacited by 90
- 2023Deep Model Fusion: A Surveycited by 89
- 2022When Do Flat Minima Optimizers Work?cited by 78
- 2019Large Scale Structure of Neural Network Loss Landscapescited by 68
- 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
- 2021Fed2: Feature-Aligned Federated Learningcited by 42
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2024: 1.
1 paper.