mlsanityData to 14 Oct 2026

Improving Language Plasticity via Pretraining with Active Forgetting

arXiv 2307.01163 ↗First posted 3 Jul 2023cs.CL cs.LG cs.NEcited by 43

Not on the map. Why?

Abstract (arXiv)

Pretrained language models (PLMs) are today the primary model for natural language processing. Despite their impressive downstream performance, it can be difficult to apply PLMs to new languages, a barrier to making their capabilities universally accessible. While prior work has shown it possible to address this issue by learning a new embedding layer for the new language, doing so is both data and compute inefficient. We propose to use an active forgetting mechanism during pretraining, as a simple way of creating PLMs that can quickly adapt to new languages. Concretely, by resetting the embedding layer every K updates during pretraining, we encourage the PLM to improve its ability of learning new embeddings within a limited number of updates, similar to a meta-learning effect. Experiments with RoBERTa show that models pretrained with our forgetting mechanism not only demonstrate faster convergence during language adaptation but also outperform standard ones in a low-data regime, particularly for languages that are distant from English.

Key facts. Cited by 43 papers in the map (data through 14 Oct 2026). Builds on 56 of its 72 references that are in the map. First posted on arXiv 3 Jul 2023.

Builds on

56 of 72 references.

References · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
38 more
  1. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  2. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,989
  3. 2017Gradient Episodic Memory for Continual Learningcited by 1,898
  4. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,565
  5. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  6. 2018Continual Lifelong Learning with Neural Networks: A Reviewcited by 1,330
  7. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  8. 2017Continual Learning with Deep Generative Replaycited by 1,158
  9. 2018Experience Replay for Continual Learningcited by 1,095
  10. 2021Ethical and social risks of harm from Language Modelscited by 966
  11. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  12. 2017PackNet: Adding Multiple Tasks to a Single Network by Iterative Pruningcited by 898
  13. 2016Learning to learn by gradient descent by gradient descentcited by 620
  14. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  15. 2018Progress & Compress: A scalable framework for continual learningcited by 520
  16. 2021Fast Model Editing at Scalecited by 498
  17. 2022Memory-Based Model Editing at Scalecited by 447
  18. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 445
  19. 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 340
  20. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  21. 2020Towards Continual Reinforcement Learning: A Review and Perspectivescited by 274
  22. 2021Scaling Laws for Transfercited by 272
  23. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  24. 2023Understanding plasticity in neural networkscited by 174
  25. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  26. 2022Adapting Pre-trained Language Models to African Languages via Multilingual Adaptive Fine-Tuningcited by 149
  27. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  28. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  29. 2020UNKs Everywhere: Adapting Multilingual Language Models to New Scriptscited by 113
  30. 2020Efficient Continual Learning with Modular Networks and Task-Driven Priorscited by 88
  31. 2020Learning to Continually Learncited by 84
  32. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  33. 2020Transient Non-Stationarity and Generalisation in Deep Reinforcement Learningcited by 75
  34. 2023Deep Reinforcement Learning with Plasticity Injectioncited by 74
  35. 2021AmericasNLI: Evaluating Zero-shot Natural Language Understanding of Pretrained Multilingual Models in Truly Low-resource Languagescited by 50
  36. 2022Fortuitous Forgetting in Connectionist Networkscited by 39
  37. 2022MasakhaNER 2.0: Africa-centric Transfer Learning for Named Entity Recognitioncited by 38
  38. 2022Building a Subspace of Policies for Scalable Continual Learningcited by 28

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 19; 2025: 14; 2026: 8.

43 papers.

Sort:

By area:

  1. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
  2. 2025SuperBPE: Space Travel for Language Modelscited by 63
  3. 2023OFA: A Framework of Initializing Unseen Subword Embeddings for Efficient Large-scale Multilingual Continued Pretrainingcited by 37
  4. 2024Zero-Shot Tokenizer Transfercited by 31
  5. 2024MaLA-500: Massive Language Adaptation of Large Language Modelscited by 28
  6. 2025Scaling Embedding Layers in Language Modelscited by 20
  7. 2024Understanding and Mitigating Tokenization Bias in Language Modelscited by 18
  8. 2025Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMscited by 18
  9. 2024Investigating Critical Period Effects in Language Acquisition through Neural Language Modelscited by 16
  10. 2024A Survey to Recent Progress Towards Understanding In-Context Learningcited by 14
  11. 2025Understanding Language Prior of LVLMs by Contrasting Chain-of-Embeddingcited by 14
  12. 2024Co-occurrence is not Factual Association in Language Modelscited by 12
31 more
  1. 2025One Tokenizer To Rule Them All: Emergent Language Plasticity via Multilingual Tokenizerscited by 12
  2. 2024The Role of Language Imbalance in Cross-lingual Generalisation: Insights from Cloned Language Experimentscited by 9
  3. 2025UltraEdit: Training-, Subject-, and Memory-Free Lifelong Editing in Language Modelscited by 9
  4. 2024Demystifying Language Model Forgetting with Low-rank Example Associationscited by 8
  5. 2024AsEP: Benchmarking Deep Learning Methods for Antibody-specific Epitope Predictioncited by 8
  6. 2024Facilitating large language model Russian adaptation with Learned Embedding Propagationcited by 8
  7. 2024DASH: Warm-Starting Neural Network Training in Stationary Settings without Loss of Plasticitycited by 7
  8. 2025Sampling from Your Language Model One Byte at a Timecited by 7
  9. 2025Continually Adding New Languages to Multilingual Language Modelscited by 7
  10. 2023Reset It and Forget It: Relearning Last-Layer Weights Improves Continual and Transfer Learningcited by 6
  11. 2024How Data Inter-connectivity Shapes LLMs Unlearning: A Structural Unlearning Perspectivecited by 5
  12. 2024DEPT: Decoupled Embeddings for Pre-training Language Modelscited by 5
  13. 2025Breaking Physical and Linguistic Borders: Multilingual Federated Prompt Tuning for Low-Resource Languagescited by 5
  14. 2026Weight Decay Improves Language Model Plasticitycited by 4
  15. 2024MAPLE: Multilingual Evaluation of Parameter Efficient Finetuning of Large Language Modelscited by 3
  16. 2026Rethinking Data Curation in LLM Training: Online Reweighting Offers Better Generalization than Offline Methodscited by 3
  17. 2024Exploring Pretraining via Active Forgetting for Improving Cross Lingual Transfer for Decoder Language Modelscited by 2
  18. 2025Vocabulary embeddings organize linguistic structure early in language model trainingcited by 2
  19. 2026AlienLM: Alienization of Language for API-Boundary Privacy in Black-Box LLMscited by 2
  20. 2026Dispersion Loss Counteracts Embedding Condensation and Improves Generalization in Small Language Modelscited by 2
  21. 2025Graceful Forgetting in Generative Language Modelscited by 1
  22. 2025How Weight Resampling and Optimizers Shape the Dynamics of Continual Learning and Forgetting in Neural Networkscited by 1
  23. 2025Multilingual Self-Taught Faithfulness Evaluatorscited by 1
  24. 2025SpidR-Adapt: A Universal Speech Representation Model for Few-Shot Adaptationcited by 1
  25. 2026To See the Unseen: on the Generalization Ability of Transformers in Symbolic Reasoningcited by 1
  26. 2026TokAlign++: Advancing Vocabulary Adaptation via Better Token Alignmentcited by 1
  27. 2024KPC-cF: Aspect-Based Sentiment Analysis via Implicit-Feature Alignment with Corpus Filteringcited by 0
  28. 2024Decoupled Vocabulary Learning Enables Zero-Shot Translation from Unseen Languagescited by 0
  29. 2024Exploring Continual Fine-Tuning for Enhancing Language Ability in Large Language Modelcited by 0
  30. 2026Memory as a Markov Matrix: Sample Efficient Knowledge Expansion via Token-to-Dictionary Mappingcited by 0
  31. 2026Learning Dynamics of Continual Learning: A Unified View of Data Attribution, Forgetting, and Plasticity Losscited by 0