mlsanityData to 7 Oct 2026

G-MAP: General Memory-Augmented Pre-trained Language Model for Domain Tasks

arXiv 2212.03613 ↗First posted 7 Dec 2022cs.CLcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, domain-specific PLMs have been proposed to boost the task performance of specific domains (e.g., biomedical and computer science) by continuing to pre-train general PLMs with domain-specific corpora. However, this Domain-Adaptive Pre-Training (DAPT; Gururangan et al. (2020)) tends to forget the previous general knowledge acquired by general PLMs, which leads to a catastrophic forgetting phenomenon and sub-optimal performance. To alleviate this problem, we propose a new framework of General Memory Augmented Pre-trained Language Model (G-MAP), which augments the domain-specific PLM by a memory representation built from the frozen general PLM without losing any general knowledge. Specifically, we propose a new memory-augmented layer, and based on it, different augmented strategies are explored to build the memory representation and then adaptively fuse it into the domain-specific PLM. We demonstrate the effectiveness of G-MAP on various domains (biomedical and computer science publications, news, and reviews) and different kinds (text classification, QA, NER) of tasks, and the extensive results show that the proposed G-MAP can achieve SOTA results on all tasks.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 41 of its 72 references that are in the map. First posted on arXiv 7 Dec 2022.

Builds on

41 of 72 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  8. 2016Learning without Forgettingcited by 2,465
  9. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
29 more
  1. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  2. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  3. 2018Efficient Neural Architecture Search via Parameter Sharingcited by 831
  4. 2013An Empirical Investigation of Catastrophic Forgetting in Gradient-Based Neural Networkscited by 711
  5. 2015Image-based Recommendations on Styles and Substitutescited by 702
  6. 2018Overcoming catastrophic forgetting with hard attention to the taskcited by 672
  7. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  8. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  9. 2021Gradient Projection Memory for Continual Learningcited by 338
  10. 2016NewsQA: A Machine Comprehension Datasetcited by 330
  11. 2018Multi-Task Identification of Entities, Relations, and Coreference for Scientific Knowledge Graph Constructioncited by 321
  12. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 255
  13. 2019ScispaCy: Fast and Robust Models for Biomedical Natural Language Processingcited by 244
  14. 2022Memorizing Transformerscited by 219
  15. 2020Recall and Learn: Fine-tuning Deep Pretrained Language Models with Less Forgettingcited by 169
  16. 2019Mixout: Effective Regularization to Finetune Large-scale Pretrained Language Modelscited by 141
  17. 2018emrQA: A Large Corpus for Question Answering on Electronic Medical Recordscited by 134
  18. 2017PubMed 200k RCT: a Dataset for Sequential Sentence Classification in Medical Abstractscited by 123
  19. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 71
  20. 2022Training Data is More Valuable than You Think: A Simple and Effective Method by Retrieving from Training Datacited by 61
  21. 2022Standing on the Shoulders of Giant Frozen Language Modelscited by 33
  22. 2021Fine-Tuned Transformers Show Clusters of Similar Representations Across Layerscited by 29
  23. 2021Importance-based Neuron Allocation for Multilingual Neural Machine Translationcited by 23
  24. 2022KALA: Knowledge-Augmented Language Model Adaptationcited by 21
  25. 2022Self-consistent Reasoning For Solving Math Word Problemscited by 10
  26. 2020Large Scale Many-Objective Optimization Driven by Distributional Adversarial Networkscited by 6
  27. 2020Many-Objective Estimation of Distribution Optimization Algorithm Based on WGAN-GPcited by 6
  28. 2021Task-adaptive Pre-training of Language Models with Word Embedding Regularizationcited by 5
  29. 2021Back-Translated Task Adaptive Pretraining: Improving Accuracy and Robustness on Text Classificationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 4; 2025: 2.

9 papers.

  1. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  2. 2023Domain Specialization as the Key to Make Large Language Models Disruptive: A Comprehensive Surveycited by 134
  3. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  4. 2023Med-UniC: Unifying Cross-Lingual Medical Vision-Language Pre-Training by Diminishing Biascited by 57
  5. 2024D2O: Dynamic Discriminative Operations for Efficient Long-Context Inference of Large Language Modelscited by 30
  6. 2024MEIT: Multimodal Electrocardiogram Instruction Tuning on Large Language Models for Report Generationcited by 29
  7. 2025Robustness in Large Language Models: A Survey of Mitigation Strategies and Evaluation Metricscited by 12
  8. 2024Retrieval-Enhanced Machine Learning: Synthesis and Opportunitiescited by 11
  9. 2025Enhancing Test-Time Scaling of Large Language Models with Hierarchical Retrieval-Augmented MCTScited by 4