mlsanityData to 7 Oct 2026

REPAIR: REnormalizing Permuted Activations for Interpolation Repair

arXiv 2211.08403 ↗First posted 15 Nov 2022cs.LG cs.AI cs.CVcited by 119

Not on the map. Why?

Abstract (arXiv)

In this paper we look into the conjecture of Entezari et al. (2021) which states that if the permutation invariance of neural networks is taken into account, then there is likely no loss barrier to the linear interpolation between SGD solutions. First, we observe that neuron alignment methods alone are insufficient to establish low-barrier linear connectivity between SGD solutions due to a phenomenon we call variance collapse: interpolated deep networks suffer a collapse in the variance of their activations, causing poor performance. Next, we propose REPAIR (REnormalizing Permuted Activations for Interpolation Repair) which mitigates variance collapse by rescaling the preactivations of such interpolated networks. We explore the interaction between our method and the choice of normalization layer, network width, and depth, and demonstrate that using REPAIR on top of neuron alignment methods leads to 60%-100% relative barrier reduction across a wide variety of architecture families and tasks. In particular, we report a 74% barrier reduction for ResNet50 on ImageNet and 90% barrier reduction for ResNet18 on CIFAR10.

Key facts. Cited by 119 papers in the map (data through 7 Oct 2026). Builds on 40 of its 54 references that are in the map. First posted on arXiv 15 Nov 2022.

Builds on

40 of 54 references.

References · 40

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2016Layer Normalizationcited by 5,579
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  9. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  10. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  11. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  12. 2021Robust fine-tuning of zero-shot modelscited by 785
28 more
  1. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  2. 2017Exploring Generalization in Deep Learningcited by 595
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  4. 2020Federated Learning with Matched Averagingcited by 564
  5. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  6. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  7. 2019Deep Ensembles: A Loss Landscape Perspectivecited by 438
  8. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  9. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 285
  10. 2020Loss landscapes and optimization in over-parameterized non-linear systems and neural networkscited by 231
  11. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 223
  12. 2019Model Fusion via Optimal Transportcited by 217
  13. 2015Convergent Learning: Do different neural networks learn the same representations?cited by 215
  14. 2019Fixup Initialization: Residual Learning Without Normalizationcited by 194
  15. 2022Patching open-vocabulary models by interpolating weightscited by 187
  16. 2021Geometry of the Loss Landscape in Overparameterized Neural Networks: Symmetries and Invariancescited by 125
  17. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  18. 2020Optimizing Mode Connectivity via Neuron Alignmentcited by 85
  19. 2018The jamming transition as a paradigm to understand the loss landscape of deep neural networkscited by 61
  20. 2019Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscapecited by 61
  21. 2022Linear Connectivity Reveals Generalization Strategiescited by 55
  22. 2019Shaping the learning landscape in neural networks around wide flat minimacited by 47
  23. 2018On the loss landscape of a class of deep neural networks with no bad local valleyscited by 38
  24. 2022Deep Networks on Toroids: Removing Symmetries Reveals the Structure of Flat Regions in the Landscape Geometrycited by 27
  25. 2022Stochastic Weight Averaging Revisitedcited by 26
  26. 2020Entropic gradient descent algorithms and wide flat minimacited by 25
  27. 2018Multi-Task Zipping via Layer-wise Neuron Sharingcited by 20
  28. 2020Safe Crossover of Neural Networks Through Neuron Alignmentcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 26; 2024: 40; 2025: 27; 2026: 23.

119 papers.

Sort:

By area:

  1. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  2. 2024The Platonic Representation Hypothesiscited by 362
  3. 2024Model Merging in LLMs, MLLMs, and Beyond: Methods, Theories, Applications and Opportunitiescited by 247
  4. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 189
  5. 2023ZipIt! Merging Models from Different Tasks without Trainingcited by 178
  6. 2024Task Singular Vectors: Reducing Task Interference in Model Mergingcited by 143
  7. 2024Model merging with SVD to tie the Knotscited by 118
  8. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  9. 2023Merge, Then Compress: Demystify Efficient SMoE with Hints from Its Routing Policycited by 104
  10. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  11. 2023Deep Model Fusion: A Surveycited by 89
  12. 2022Re-basin via implicit Sinkhorn differentiationcited by 61
38 more
  1. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  2. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  3. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  4. 2024What Matters for Model Merging at Scale?cited by 52
  5. 2023UnIVAL: Unified Model for Image, Video, Audio and Language Taskscited by 44
  6. 2024Merge, Ensemble, and Cooperate! A Survey on Collaborative Strategies in the Era of Large Language Modelscited by 43
  7. 2023Hidden symmetries of ReLU networkscited by 38
  8. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  9. 2023Equivariant Deep Weight Space Alignmentcited by 34
  10. 2024Training-Free Pretrained Model Mergingcited by 32
  11. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  12. 2025Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunchcited by 26
  13. 2024Asynchronous Local-SGD Training for Language Modelingcited by 24
  14. 2023PopulAtion Parameter Averaging (PAPA)cited by 23
  15. 2023Layer-wise Linear Mode Connectivitycited by 22
  16. 2024Multi-Task Model Merging via Adaptive Weight Disentanglementcited by 22
  17. 2024FreeStyle: Free Lunch for Text-guided Style Transfer using Diffusion Modelscited by 21
  18. 2024The Empirical Impact of Neural Parameter Symmetries, or Lack Thereofcited by 21
  19. 2023Sparse Model Soups: A Recipe for Improved Pruning via Model Averagingcited by 20
  20. 2024C2M3: Cycle-Consistent Multi-Model Mergingcited by 20
  21. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  22. 2025Update Your Transformer to the Latest Release: Re-Basin of Task Vectorscited by 17
  23. 2024PLeaS -- Merging Models with Permutations and Least Squarescited by 16
  24. 2025Sens-Merging: Sensitivity-Guided Parameter Balancing for Merging Large Language Modelscited by 16
  25. 2023Merging Decision Transformers: Weight Averaging for Forming Multi-Task Policiescited by 15
  26. 2024Mind the Modality Gap: Towards a Remote Sensing Vision-Language Model via Cross-modal Alignmentcited by 15
  27. 2024FedFisher: Leveraging Fisher Information for One-Shot Federated Learningcited by 15
  28. 2024Training-Free Model Merging for Multi-target Domain Adaptationcited by 15
  29. 2025Generalized Linear Mode Connectivity for Transformerscited by 14
  30. 2023Fuse to Forget: Bias Reduction and Selective Memorization through Model Fusioncited by 13
  31. 2024Continual Learning with Weight Interpolationcited by 13
  32. 2024Realistic Evaluation of Model Merging for Compositional Generalizationcited by 12
  33. 2024SurgeryV2: Bridging the Gap Between Model Merging and Multi-Task Learning with Deep Representation Surgerycited by 12
  34. 2024ATM: Improving Model Merging by Alternating Tuning and Mergingcited by 10
  35. 2025Understanding Mode Connectivity via Parameter Space Symmetrycited by 10
  36. 2024Simultaneous linear connectivity of neural networks modulo permutationcited by 9
  37. 2024MaxFusion: Plug&Play Multi-Modal Generation in Text-to-Image Diffusion Modelscited by 9
  38. 2024Harmony in Diversity: Merging Neural Networks with Canonical Correlation Analysiscited by 9

50 most cited of 119 All 119 (JSON)