Data to 7 Oct 2026

Better Fine-Tuning by Reducing Representational Collapse

arXiv 2008.03156 ↗First posted 6 Aug 2020cs.LG cs.CL stat.MLcited by 156

Not on the map. Why?

Abstract (arXiv)

Although widely adopted, existing approaches for fine-tuning pre-trained language models have been shown to be unstable across hyper-parameter settings, motivating recent work on trust region methods. In this paper, we present a simplified and efficient method rooted in trust region theory that replaces previously used adversarial objectives with parametric noise (sampling from either a normal or uniform distribution), thereby discouraging representation change during fine-tuning when possible without hurting performance. We also introduce a new analysis to motivate the use of trust region methods more generally, by studying representational collapse; the degradation of generalizable representations from pre-trained models as they are fine-tuned for a specific end task. Extensive experiments show that our fine-tuning method matches or exceeds the performance of previous trust region methods on a range of understanding and generation tasks (including DailyMail/CNN, Gigaword, Reddit TIFU, and the GLUE benchmark), while also being much faster. We also show that it is less prone to representation collapse; the pre-trained models maintain more generalizable representations every time they are fine-tuned.

Key facts. Cited by 156 papers in the map (data through 7 Oct 2026). First posted on arXiv 6 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 29; 2022: 42; 2023: 34; 2024: 23; 2025: 15; 2026: 13.

156 papers.

Sort:

By area:

2021Robust fine-tuning of zero-shot modelscited by 785
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
2022UL2: Unifying Language Learning Paradigmscited by 292
2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
2021R-Drop: Regularized Dropout for Neural Networkscited by 202
2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
2022On the Representation Collapse of Sparse Mixture of Expertscited by 156
2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145

38 more

2022CM3: A Causal Masked Multimodal Model of the Internetcited by 142
2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 130
2021COCO-LM: Correcting and Contrasting Text Sequences for Language Model Pretrainingcited by 126
2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
2022When Do Flat Minima Optimizers Work?cited by 78
2022Machine Generated Text: A Comprehensive Survey of Threat Models and Detection Methodscited by 75
2022SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarizationcited by 61
2021HTLM: Hyper-Text Pre-Training and Prompting of Language Modelscited by 58
2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
2025The Best Instruction-Tuning Data are Those That Fitcited by 48
2024Bridging Remote Sensors with Multisensor Geospatial Foundation Modelscited by 46
2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
2024LiNeS: Post-training Layer Scaling Prevents Forgetting and Enhances Model Mergingcited by 39
2021Improved Fine-Tuning by Better Leveraging Pre-Training Datacited by 36
2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 31
2021Sequence Level Contrastive Learning for Text Summarizationcited by 31
2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
2023On the Embedding Collapse when Scaling up Recommendation Modelscited by 29
2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
2022Exploring The Landscape of Distributional Robustness for Question Answering Modelscited by 23
2022How to Fine-Tune Vision Models with SGDcited by 23
2023Improving Generalization of Adversarial Training via Robust Critical Fine-Tuningcited by 23
2021Reinforcement Learning for Abstractive Question Summarization with Question-aware Semantic Rewardscited by 21
2022Adapting Pretrained Text-to-Text Models for Long Text Sequencescited by 21
2024Securely Fine-tuning Pre-trained Encoders Against Adversarial Examplescited by 21
2023Weakly Supervised Detection of Hallucinations in LLM Activationscited by 20
2021Span Pointer Networks for Non-Autoregressive Task-Oriented Semantic Parsingcited by 18
2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 18
2022Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptivelycited by 18
2022BARTSmiles: Generative Masked Language Models for Molecular Representationscited by 18
2023How to prepare your task head for finetuningcited by 18
2025Large language models for artificial general intelligence (AGI): A survey of foundational principles and approachescited by 18

50 most cited of 156 All 156 (JSON)