mlsanityData to 7 Oct 2026

AdapterSwap: Continuous Training of LLMs with Data Removal and Access-Control Guarantees

arXiv 2404.08417 ↗First posted 12 Apr 2024cs.LG cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) are increasingly capable of completing knowledge intensive tasks by recalling information from a static pretraining corpus. Here we are concerned with LLMs in the context of evolving data requirements. For instance: batches of new data that are introduced periodically; subsets of data with user-based access controls; or requirements on dynamic removal of documents with guarantees that associated knowledge cannot be recalled. We wish to satisfy these requirements while at the same time ensuring a model does not forget old information when new data becomes available. To address these issues, we introduce AdapterSwap, a training and inference scheme that organizes knowledge from a data collection into a set of low-rank adapters, which are dynamically composed during inference. Our experiments demonstrate AdapterSwap's ability to support efficient continual learning, while also enabling organizations to have fine-grained control over data access and deletion.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 36 of its 48 references that are in the map. First posted on arXiv 12 Apr 2024.

Builds on

36 of 48 references.

References · 36

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  11. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  12. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
24 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  3. 2022Quantifying Memorization Across Neural Language Modelscited by 755
  4. 2023The RefinedWeb Dataset for Falcon LLM: Outperforming Curated Corpora with Web Data, and Web Data Onlycited by 755
  5. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 622
  6. 2021Editing Factual Knowledge in Language Modelscited by 548
  7. 2023Detecting Pretraining Data from Large Language Modelscited by 375
  8. 2023Editing Large Language Models: Problems, Methods, and Opportunitiescited by 367
  9. 2022The Stack: 3 TB of permissively licensed source codecited by 354
  10. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  11. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  12. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  13. 2022Memorization Without Overfitting: Analyzing the Training Dynamics of Large Language Modelscited by 218
  14. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  15. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  16. 2023Inspecting and Editing Knowledge Representations in Language Modelscited by 140
  17. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 138
  18. 2022AdaMix: Mixture-of-Adaptations for Parameter-efficient Model Tuningcited by 137
  19. 2022StreamingQA: A Benchmark for Adaptation to New Knowledge over Time in Question Answering Modelscited by 123
  20. 2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
  21. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  22. 2022When Federated Learning Meets Pre-trained Language Models' Parameter-Efficient Tuning Methodscited by 74
  23. 2022Recovering Private Text in Federated Learning of Language Modelscited by 67
  24. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025Training Plug-n-Play Knowledge Modules with Deep Context Distillationcited by 22
  2. 2025FlexOlmo: Open Language Models for Flexible Data Usecited by 14
  3. 2024RE-Adapt: Reverse Engineered Adaptation of Large Language Modelscited by 9
  4. 2025Permissioned LLMs: Enforcing Access Control in Large Language Modelscited by 7
  5. 2025LoRA-Augmented Generation (LAG) for Knowledge-Intensive Language Taskscited by 7
  6. 2025SpectR: Dynamically Composing LM Experts with Spectral Routingcited by 6
  7. 2025Role-Conditioned Refusals: Evaluating Access Control Reasoning in Large Language Modelscited by 4
  8. 2025MegaWika 2: A More Comprehensive Multilingual Collection of Articles and their Sourcescited by 3
  9. 2025Role-Aware Language Models for Secure and Contextualized Access Control in Organizationscited by 2
  10. 2025SEQR: Secure and Efficient QR-based LoRA Routingcited by 1
  11. 2026PermDoRA -- Understanding Adapter Interference in Language Models: Limits of Parameter-Space Geometrycited by 0
  12. 2026Toward Open Weight Models Without Risks: Separating Public and Private Capabilities in LLMscited by 0
2 more
  1. 2026Benchmarking Text-to-SQL under Role-Based Access Controlcited by 0
  2. 2026Knowledge Pull Requests for Continual Document Authoringcited by 0