mlsanityData to 7 Oct 2026

Exploring Mode Connectivity for Pre-trained Language Models

arXiv 2210.14102 ↗First posted 25 Oct 2022cs.CL cs.AIcited by 26

Not on the map. Why?

Abstract (arXiv)

Recent years have witnessed the prevalent application of pre-trained language models (PLMs) in NLP. From the perspective of parameter space, PLMs provide generic initialization, starting from which high-performance minima could be found. Although plenty of works have studied how to effectively and efficiently adapt PLMs to high-performance minima, little is known about the connection of various minima reached under different adaptation configurations. In this paper, we investigate the geometric connections of different minima through the lens of mode connectivity, which measures whether two minima can be connected with a low-loss path. We conduct empirical analyses to investigate three questions: (1) how could hyperparameters, specific tuning methods, and training data affect PLM's mode connectivity? (2) How does mode connectivity change during pre-training? (3) How does the PLM's task knowledge change along the path connecting two minima? In general, exploring the mode connectivity of PLMs conduces to understanding the geometric connection of different minima, which may help us fathom the inner workings of PLM downstream adaptation.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). Builds on 43 of its 55 references that are in the map. First posted on arXiv 25 Oct 2022.

Builds on

43 of 55 references.

References · 43

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  11. 2020Extracting Training Data from Large Language Modelscited by 2,144
  12. 2019Language Models as Knowledge Bases?cited by 1,922
31 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  5. 2021Robust fine-tuning of zero-shot modelscited by 785
  6. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  7. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  8. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  9. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  10. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  11. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  12. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  13. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  14. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  15. 2020What is being transferred in transfer learning?cited by 365
  16. 2018Essentially No Barriers in Neural Network Energy Landscapecited by 328
  17. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  18. 2020Pretrained Transformers Improve Out-of-Distribution Robustnesscited by 293
  19. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  20. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  21. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  22. 2020Linear Mode Connectivity in Multitask and Continual Learningcited by 144
  23. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  24. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  25. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
  26. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  27. 2019Visualizing and Understanding the Effectiveness of BERTcited by 67
  28. 2021What to Pre-Train on? Efficient Intermediate Task Selectioncited by 62
  29. 2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
  30. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  31. 2021An Empirical Study on Hyperparameter Optimization for Fine-Tuning Pre-trained Language Modelscited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 9; 2024: 4; 2025: 7; 2026: 4.

26 papers.

Sort:

By area:

  1. 2023Language Models are Super Mario: Absorbing Abilities from Homologous Models as a Free Lunchcited by 679
  2. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
  3. 2023Composing Parameter-Efficient Modules with Arithmetic Operationscited by 155
  4. 2022Model Ratatouille: Recycling Diverse Models for Out-of-Distribution Generalizationcited by 97
  5. 2023Deep Model Fusion: A Surveycited by 89
  6. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  7. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  8. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  9. 2023Merging by Matching Models in Task Parameter Subspacescited by 36
  10. 2024On the Emergence of Cross-Task Linearity in the Pretraining-Finetuning Paradigmcited by 25
  11. 2023Proving Linear Mode Connectivity of Neural Networks via Optimal Transportcited by 24
  12. 2023Recyclable Tuning for Continual Pre-trainingcited by 17
14 more
  1. 2024Training-Free Model Merging for Multi-target Domain Adaptationcited by 15
  2. 2025LLM Unlearning Reveals a Stronger-Than-Expected Coreset Effect in Current Benchmarkscited by 11
  3. 2023Lookaround Optimizer: k steps around, 1 step averagecited by 9
  4. 2024DHA: Learning Decoupled-Head Attention from Transformer Checkpoints via Adaptive Heads Fusioncited by 6
  5. 2025Downgrade to Upgrade: Optimizer Simplification Enhances Robustness in LLM Unlearningcited by 4
  6. 2026An Empirical Study and Theoretical Explanation on Task-Level Model-Merging Collapsecited by 4
  7. 2025Unveiling Mode Connectivity in Graph Neural Networkscited by 3
  8. 2025DitHub: A Modular Framework for Incremental Open-Vocabulary Object Detectioncited by 3
  9. 2026Decouple Searching from Training: Scaling Data Mixing via Model Merging for Large Language Model Pre-trainingcited by 2
  10. 2026Decentralized Instruction Tuning: Conflict-Aware Splitting and Weight Mergingcited by 2
  11. 2025Robustness and Regularization in Hierarchical Re-Basincited by 1
  12. 2024Manifold Metric: A Loss Landscape Approach for Predicting Model Performancecited by 0
  13. 2025Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 0
  14. 2026Understanding Machine Unlearning Through the Lens of Mode Connectivitycited by 0