mlsanityData to 14 Oct 2026

Adapting a Language Model While Preserving its General Knowledge

arXiv 2301.08986 ↗First posted 21 Jan 2023cs.CL cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

Domain-adaptive pre-training (or DA-training for short), also known as post-training, aims to train a pre-trained general-purpose language model (LM) using an unlabeled corpus of a particular domain to adapt the LM so that end-tasks in the domain can give improved performances. However, existing DA-training methods are in some sense blind as they do not explicitly identify what knowledge in the LM should be preserved and what should be changed by the domain corpus. This paper shows that the existing methods are suboptimal and proposes a novel method to perform a more informed adaptation of the knowledge in the LM by (1) soft-masking the attention heads based on their importance to best preserve the general knowledge in the LM and (2) contrasting the representations of the general and the full (both general and domain knowledge) to learn an integrated representation with both general and domain-specific knowledge. Experimental results will demonstrate the effectiveness of the proposed approach.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 21 Jan 2023.

Builds on

39 of 49 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  7. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  8. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  10. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  11. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  12. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
27 more
  1. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,947
  2. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  3. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,414
  4. 2019Publicly Available Clinical BERT Embeddingscited by 1,002
  5. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  6. 2019Are Sixteen Heads Really Better than One?cited by 816
  7. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  8. 2019How to Fine-Tune BERT for Text Classification?cited by 411
  9. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 322
  10. 2020The Lottery Ticket Hypothesis for Pre-trained BERT Networkscited by 275
  11. 2016Dynamic Network Surgery for Efficient DNNscited by 233
  12. 2019BERT Post-Training for Review Reading Comprehension and Aspect-based Sentiment Analysiscited by 176
  13. 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
  14. 2021Achieving Forgetting Prevention and Knowledge Transfer in Continual Learningcited by 91
  15. 2019A Mutual Information Maximization Perspective of Language Representation Learningcited by 85
  16. 2016Aspect Level Sentiment Classification with Deep Memory Networkcited by 82
  17. 2021Continual Learning of a Mixed Sequence of Similar and Dissimilar Taskscited by 81
  18. 2021PARP: Prune, Adjust and Re-Prune for Self-Supervised Speech Recognitioncited by 53
  19. 2021Differentiable Subset Pruning of Transformer Headscited by 47
  20. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  21. 2021TaCL: Improving BERT Pre-training with Token-aware Contrastive Learningcited by 32
  22. 2019IMHO Fine-Tuning Improves Claim Detectioncited by 27
  23. 2021Should We Be Pre-training? An Argument for End-task Aware Training as an Alternativecited by 26
  24. 2020Pruning Redundant Mappings in Transformer Models via Spectral-Normalized Identity Priorcited by 23
  25. 2022Contextual Representation Learning beyond Masked Language Modelingcited by 16
  26. 2021Adapting by Pruning: A Case Study on BERTcited by 11
  27. 2019Review Conversational Reading Comprehensioncited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 8; 2024: 10; 2025: 3; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023Continual Pre-training of Language Modelscited by 198
  2. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 137
  3. 2024RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Horizon Generationcited by 75
  4. 2023Continual Learning: Applications and the Road Forwardcited by 59
  5. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 57
  6. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  7. 2023Efficient Continual Pre-training for Building Domain Specific Large Language Modelscited by 34
  8. 2024Selecting Large Language Model to Fine-tune via Rectified Scaling Lawcited by 33
  9. 2024Efficient Continual Pre-training by Mitigating the Stability Gapcited by 25
  10. 2025Demystifying Domain-adaptive Post-training for Financial LLMscited by 24
  11. 2024Are LLMs Prescient? A Continuous Evaluation using Daily News as the Oraclecited by 19
  12. 2023Class-Incremental Learning based on Label Generationcited by 11
11 more
  1. 2024Aurora-M: Open Source Continual Pre-training for Multilingual Language and Codecited by 10
  2. 2023Towards Anytime Fine-tuning: Continually Pre-trained Language Models with Hypernetwork Promptcited by 8
  3. 2024BioMamba: Domain-Adaptive Biomedical Language Modelscited by 8
  4. 2024Mix-CPT: A Domain Adaptation Framework via Decoupling Knowledge Learning and Format Alignmentcited by 7
  5. 2026Beyond Static Models: An Evolving Framework for Continual Learning in Large Language Models across Training Stagescited by 6
  6. 2023Difference-Masking: Choosing What to Mask in Continued Pretrainingcited by 4
  7. 2024Word Matters: What Influences Domain Adaptation in Summarization?cited by 3
  8. 2025DoMIX: An Efficient Framework for Exploiting Domain Knowledge in Fine-Tuningcited by 2
  9. 2024Balancing the Causal Effects in Class-Incremental Learningcited by 1
  10. 2025Fragile Mastery: Are Domain-Specific Trade-Offs Undermining On-Device Language Models?cited by 0
  11. 2026Reading the News: Adapting Large Language Models to Swedish Journalism Through Continued Pre-Trainingcited by 0