Data to 7 Oct 2026

SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimization

arXiv 1911.03437 ↗First posted 8 Nov 2019cs.CL cs.LG math.OCcited by 253

Not on the map. Why?

Abstract (arXiv)

Transfer learning has fundamentally changed the landscape of natural language processing (NLP) research. Many existing state-of-the-art models are first pre-trained on a large text corpus and then fine-tuned on downstream tasks. However, due to limited data resources from downstream tasks and the extremely large capacity of pre-trained models, aggressive fine-tuning often causes the adapted model to overfit the data of downstream tasks and forget the knowledge of the pre-trained model. To address the above issue in a more principled manner, we propose a new computational framework for robust and efficient fine-tuning for pre-trained language models. Specifically, our proposed framework contains two important ingredients: 1. Smoothness-inducing regularization, which effectively manages the capacity of the model; 2. Bregman proximal point optimization, which is a class of trust-region methods and can prevent knowledge forgetting. Our experiments demonstrate that our proposed method achieves the state-of-the-art performance on multiple NLP benchmarks.

Key facts. Cited by 253 papers in the map (data through 7 Oct 2026). First posted on arXiv 8 Nov 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 61; 2022: 67; 2023: 61; 2024: 37; 2025: 17; 2026: 10.

253 papers.

Sort:

By area:

2021Robust fine-tuning of zero-shot modelscited by 785
2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 722
2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
2022Selective Annotation Makes Language Models Better Few-Shot Learnerscited by 242
2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
2021Adversarial GLUE: A Multi-Task Benchmark for Robustness Evaluation of Language Modelscited by 227
2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
2024A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systemscited by 170
2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 158
2022Finetune like you pretrain: Improved finetuning of zero-shot vision modelscited by 149
2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145

38 more

2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
2023MiniLLM: On-Policy Distillation of Large Language Modelscited by 129
2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
2021Taming Sparsely Activated Transformer with Stochastic Expertscited by 112
2021A Survey on Data Augmentation for Text Classificationcited by 111
2021Sharpness-Aware Minimization Improves Language Model Generalizationcited by 108
2021Measure and Improve Robustness in NLP Models: A Surveycited by 107
2024Efficient Adversarial Training in LLMs with Continuous Attackscited by 104
2021Self-Guided Contrastive Learning for BERT Sentence Representationscited by 103
2021An Empirical Survey of Data Augmentation for Limited Data Learning in NLPcited by 101
2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
2024Soft Prompt Threats: Attacking Safety Alignment and Unlearning in Open-Source LLMs through the Embedding Spacecited by 88
2024Defending Against Unforeseen Failure Modes with Latent Adversarial Trainingcited by 82
2021Towards Improving Adversarial Training of NLP Modelscited by 79
2023AfriSenti: A Twitter Sentiment Analysis Benchmark for African Languagescited by 68
2024Robust LLM safeguarding via refusal feature adversarial trainingcited by 60
2022Uni-Perceiver-MoE: Learning Sparse Generalist Models with Conditional MoEscited by 58
2021Fine-Tuning Large Neural Language Models for Biomedical Natural Language Processingcited by 57
2022Transferability in Deep Learning: A Surveycited by 57
2022Debiased Contrastive Learning of Unsupervised Sentence Representationscited by 56
2021CLINE: Contrastive Learning with Semantic Negative Examples for Natural Language Understandingcited by 50
2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48
2021Playing Lottery Tickets with Vision and Languagecited by 45
2021Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinesecited by 45
2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
2021UnitedQA: A Hybrid Approach for Open Domain Question Answeringcited by 42
2021Consistency Regularization for Cross-Lingual Fine-Tuningcited by 41
2023SemEval-2023 Task 12: Sentiment Analysis for African Languages (AfriSenti-SemEval)cited by 41
2021Human Parity on CommonsenseQA: Augmenting Self-Attention with External Attentioncited by 40
2023Trained on 100 million words and still in shape: BERT meets British National Corpuscited by 39
2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Reviewcited by 37
2021SGD-X: A Benchmark for Robust Generalization in Schema-Guided Dialogue Systemscited by 34
2023Cross-Modal Fine-Tuning: Align then Refinecited by 34
2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
2023Prompt Optimization via Adversarial In-Context Learningcited by 33
2021HiddenCut: Simple Data Augmentation for Natural Language Understanding with Better Generalizationcited by 31
2021Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrificecited by 30

50 most cited of 253 All 253 (JSON)