Data to 7 Oct 2026

Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Models

arXiv 1909.11299 ↗First posted 25 Sep 2019cs.LG stat.MLcited by 141

Not on the map. Why?

Abstract (arXiv)

In natural language processing, it has been observed recently that generalization could be greatly improved by finetuning a large-scale language model pretrained on a large unlabeled corpus. Despite its recent success and wide adoption, finetuning a large pretrained language model on a downstream task is prone to degenerate performance when there are only a small number of training instances available. In this paper, we introduce a new regularization technique, to which we refer as "mixout", motivated by dropout. Mixout stochastically mixes the parameters of two models. We show that our mixout technique regularizes learning to minimize the deviation from one of the two models and that the strength of regularization adapts along the optimization trajectory. We empirically evaluate the proposed mixout and its variants on finetuning a pretrained language model on downstream tasks. More specifically, we demonstrate that the stability of finetuning and the average accuracy greatly increase when we use the proposed approach to regularize finetuning of BERT on downstream tasks in GLUE.

Key facts. Cited by 141 papers in the map (data through 7 Oct 2026). First posted on arXiv 25 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 40; 2023: 30; 2024: 25; 2025: 14; 2026: 12.

141 papers.

Sort:

By area:

2023LLM+P: Empowering Large Language Models with Optimal Planning Proficiencycited by 496
2022Aging with GRACE: Lifelong Model Editing with Discrete Key-Value Adaptorscited by 267
2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
2021How Many Data Points is a Prompt Worth?cited by 197
2022MERLOT Reserve: Neural Script Knowledge through Vision and Language and Soundcited by 184
2021Well-tuned Simple Nets Excel on Tabular Datasetscited by 151
2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2021Entailment as Few-Shot Learnercited by 122

38 more

2022Pretrained Language Models for Text Generation: A Surveycited by 121
2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 119
2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
2023Bayesian Low-rank Adaptation for Large Language Modelscited by 112
2023A Survey of Safety and Trustworthiness of Large Language Models through the Lens of Verification and Validationcited by 91
2021The MultiBERTs: BERT Reproductions for Robustness Analysiscited by 89
2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 66
2024Yo'LLaVA: Your Personalized Language and Vision Assistantcited by 61
2024MyVLM: Personalizing VLMs for User-Specific Queriescited by 58
2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
2022Transferability in Deep Learning: A Surveycited by 57
2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
2021Variational Information Bottleneck for Effective Low-Resource Fine-Tuningcited by 51
2022Fine-mixing: Mitigating Backdoors in Fine-tuned Language Modelscited by 50
2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
2024Pre-trained Model Guided Fine-Tuning for Zero-Shot Adversarial Robustnesscited by 48
2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
2021A Closer Look at How Fine-tuning Changes BERTcited by 42
2022Adaptive Transformers for Robust Few-shot Cross-domain Face Anti-spoofingcited by 38
2022u-HuBERT: Unified Mixed-Modal Speech Pretraining And Zero-Shot Transfer to Unlabeled Modalitycited by 36
2023Merging Experts into One: Improving Computational Efficiency of Mixture of Expertscited by 36
2024Scaling Laws for Forgetting When Fine-Tuning Large Language Modelscited by 36
2024Mitigating Catastrophic Forgetting in Language Transfer via Model Mergingcited by 36
2021Robust Transfer Learning with Pretrained Language Models through Adapterscited by 35
2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
2024Online Merging Optimizers for Boosting Rewards and Mitigating Tax in Alignmentcited by 28
2024Generative AI in EU Law: Liability, Privacy, Intellectual Property, and Cybersecuritycited by 21
2022A Differential Entropy Estimator for Training Neural Networkscited by 20
2023Weakly Supervised Detection of Hallucinations in LLM Activationscited by 20
2021From Dense to Sparse: Contrastive Pruning for Better Pre-trained Language Model Compressioncited by 18
2022Fine-Tuning Pre-Trained Language Models Effectively by Optimizing Subnetworks Adaptivelycited by 18
2023Randomized Sparse Neural Galerkin Schemes for Solving Evolution Equations with Deep Networkscited by 18
2024HFT: Half Fine-Tuning for Large Language Modelscited by 18
2022Parameter-Efficient Tuning Makes a Good Classification Headcited by 15
2024Learning to Rebalance Multi-Modal Optimization by Adaptively Masking Subnetworkscited by 15

50 most cited of 141 All 141 (JSON)