Data to 14 Oct 2026

Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgetting

arXiv 2004.12651 ↗First posted 27 Apr 2020cs.CLcited by 170

Not on the map. Why?

Abstract (arXiv)

Deep pretrained language models have achieved great success in the way of pretraining first and then fine-tuning. But such a sequential transfer learning paradigm often confronts the catastrophic forgetting problem and leads to sub-optimal performance. To fine-tune with less forgetting, we propose a recall and learn mechanism, which adopts the idea of multi-task learning and jointly learns pretraining tasks and downstream tasks. Specifically, we propose a Pretraining Simulation mechanism to recall the knowledge from pretraining tasks without data, and an Objective Shifting mechanism to focus the learning on downstream tasks gradually. Experiments show that our method achieves state-of-the-art performance on the GLUE benchmark. Our method also enables BERT-base to achieve better performance than directly fine-tuning of BERT-large. Further, we provide the open-source RecAdam optimizer, which integrates the proposed mechanisms into Adam optimizer, to facility the NLP community.

Key facts. Cited by 170 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 20; 2022: 34; 2023: 39; 2024: 46; 2025: 24; 2026: 7.

170 papers.

Sort:

By area:

2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 234
2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 220
2021A Survey of Human-in-the-loop for Machine Learningcited by 215
2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 206
2023Knowledge Editing for Large Language Models: A Surveycited by 194
2023Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMscited by 178
2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 175
2021Towards Continual Knowledge Learning of Language Modelscited by 174
2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 148
2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 146

38 more

2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 135
2023DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Modelscited by 128
2022TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Modelscited by 119
2023Investigating the Catastrophic Forgetting in Multimodal Large Language Modelscited by 119
2023The CoT Collection: Improving Zero-shot and Few-shot Learning of Language Models via Chain-of-Thought Fine-Tuningcited by 108
2024ChartAssisstant: A Universal Chart Multimodal Language Model via Chart-to-Table Pre-training and Multitask Instruction Tuningcited by 79
2024LLMs are Good Sign Language Translatorscited by 77
2023A Comprehensive Survey of Forgetting in Deep Learning Beyond Continual Learningcited by 69
2024Towards Lifelong Learning of Large Language Models: A Surveycited by 64
2023Lawyer LLaMA Technical Reportcited by 54
2024Model Tailor: Mitigating Catastrophic Forgetting in Multi-modal Large Language Modelscited by 54
2022Perturbation Augmentation for Fairer NLPcited by 52
2021AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarizationcited by 51
2024Addressing Loss of Plasticity and Catastrophic Forgetting in Continual Learningcited by 50
2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
2023LM-Cocktail: Resilient Tuning of Language Models via Model Mergingcited by 44
2022NoisyTune: A Little Noise Can Help You Finetune Pretrained Language Models Bettercited by 43
2021A Closer Look at How Fine-tuning Changes BERTcited by 42
2022Contrastive Learning Reduces Hallucination in Conversationscited by 40
2021Joint Unsupervised and Supervised Training for Multilingual ASRcited by 39
2023The Janus Interface: How Fine-Tuning in Large Language Models Amplifies the Privacy Riskscited by 36
2024Unveiling the Generalization Power of Fine-Tuned Large Language Modelscited by 36
2024Multi-hop Question Answering under Temporal Knowledge Editingcited by 36
2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptationcited by 31
2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
2023Preserving Pre-trained Features Helps Calibrate Fine-tuned Language Modelscited by 29
2023UniTabE: A Universal Pretraining Protocol for Tabular Foundation Model in Data Sciencecited by 29
2024Enhancing Reasoning Capabilities of LLMs via Principled Synthetic Logic Corpuscited by 29
2022On the Impossible Safety of Large AI Modelscited by 28
2023Parameter-efficient Modularised Bias Mitigation via AdapterFusioncited by 25
2024Parameter Efficient Fine Tuning: A Comprehensive Analysis Across Applicationscited by 25
2024Propagation and Pitfalls: Reasoning-based Assessment of Knowledge Editing through Counterfactual Taskscited by 24
2021Avoiding Inference Heuristics in Few-shot Prompt-based Finetuningcited by 23
2022Where to start? Analyzing the potential value of intermediate modelscited by 23
2024Explainable Generative AI (GenXAI): A Survey, Conceptualization, and Research Agendacited by 22
2022KALA: Knowledge-Augmented Language Model Adaptationcited by 21

50 most cited of 170 All 170 (JSON)