mlsanityData to 7 Oct 2026

DEMix Layers: Disentangling Domains for Modular Language Modeling

arXiv 2108.05036 ↗First posted 11 Aug 2021cs.CL cs.AIcited by 130

Not on the map. Why?

Cited by 42 in Mixture-of-experts, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We introduce a new domain expert mixture (DEMix) layer that enables conditioning a language model (LM) on the domain of the input text. A DEMix layer is a collection of expert feedforward networks, each specialized to a domain, that makes the LM modular: experts can be mixed, added or removed after initial training. Extensive experiments with autoregressive transformer LMs (up to 1.3B parameters) show that DEMix layers reduce test-time perplexity, increase training efficiency, and enable rapid adaptation with little overhead. We show that mixing experts during inference, using a parameter-free weighted ensemble, allows the model to better generalize to heterogeneous or unseen domains. We also show that experts can be added to iteratively incorporate new domains without forgetting older ones, and that experts can be removed to restrict access to unwanted domains, without additional training. Overall, these results demonstrate benefits of explicitly conditioning on textual domains during language modeling.

Key facts. Cited by 130 papers in the map (data through 7 Oct 2026). Builds on 53 of its 84 references that are in the map. First posted on arXiv 11 Aug 2021.

Builds on

53 of 84 references.

References · 53

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2017Gradient Episodic Memory for Continual Learningcited by 1,871
  11. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  12. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
38 more
  1. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  2. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  3. 2016Progressive Neural Networkscited by 1,332
  4. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  7. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  8. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  9. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 621
  10. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  11. 2019Defending Against Neural Fake Newscited by 553
  12. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  13. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 475
  14. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  15. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  16. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 315
  17. 2013One Billion Word Benchmark for Measuring Progress in Statistical Language Modelingcited by 302
  18. 2020The Pushshift Reddit Datasetcited by 289
  19. 2017Meta Networkscited by 268
  20. 2019Massively Multilingual Neural Machine Translation in the Wild: Findings and Challengescited by 253
  21. 2021Hash Layers For Large Sparse Modelscited by 236
  22. 2021CUAD: An Expert-Annotated NLP Dataset for Legal Contract Reviewcited by 232
  23. 2019Episodic Memory in Lifelong Language Learningcited by 188
  24. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  25. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  26. 2020Unsupervised Domain Clusters in Pretrained Language Modelscited by 160
  27. 2019LAMOL: LAnguage MOdeling for Lifelong Language Learningcited by 156
  28. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  29. 2019Distributionally Robust Language Modelingcited by 142
  30. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 142
  31. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  32. 2020The Radicalization Risks of GPT-3 and Advanced Neural Language Modelscited by 76
  33. 2018Predicting Factuality of Reporting and Bias of News Media Sourcescited by 71
  34. 2018Multi-Source Domain Adaptation with Mixture of Expertscited by 62
  35. 2017Neural Models for Documents with Metadatacited by 46
  36. 2016What to do about non-standard (or non-canonical) language in NLPcited by 35
  37. 2016Neurogenesis Deep Learningcited by 32
  38. 2020Transformer Based Multi-Source Domain Adaptationcited by 23

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 19; 2023: 30; 2024: 36; 2025: 27; 2026: 14.

130 papers.

Sort:

By area:

  1. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 391
  2. 2024Continual Learning of Large Language Models: A Comprehensive Surveycited by 228
  3. 2022Sparse Upcycling: Training Mixture-of-Experts from Dense Checkpointscited by 207
  4. 2023Continual Pre-training of Language Modelscited by 194
  5. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  6. 2022A Review of Sparse Expert Models in Deep Learningcited by 147
  7. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 145
  8. 2023Continual Pre-Training of Large Language Models: How to (re)warm your model?cited by 135
  9. 2021Lifelong Pretraining: Continually Adapting Language Models to Emerging Corporacited by 114
  10. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 110
  11. 2024Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLMcited by 106
  12. 2021Time Waits for No One! Analysis and Challenges of Temporal Misalignmentcited by 91
38 more
  1. 2023Soft Merging of Experts with Adaptive Routingcited by 88
  2. 2023AdapterSoup: Weight Averaging to Improve Generalization of Pretrained Language Modelscited by 84
  3. 2022Continual Learning of Natural Language Processing Tasks: A Surveycited by 83
  4. 2023Modular Deep Learningcited by 82
  5. 2023Test-Time Training on Nearest Neighbors for Large Language Modelscited by 82
  6. 2023SILO Language Models: Isolating Legal Risk In a Nonparametric Datastorecited by 71
  7. 2022ELLE: Efficient Lifelong Pre-training for Emerging Datacited by 70
  8. 2022Z-ICL: Zero-Shot In-Context Learning with Pseudo-Demonstrationscited by 70
  9. 2024In-context Learning with Retrieved Demonstrations for Language Models: A Surveycited by 69
  10. 2023Lifelong Language Pretraining with Distribution-Specialized Expertscited by 64
  11. 2024MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learningcited by 63
  12. 2022NLPeer: A Unified Resource for the Computational Study of Peer Reviewcited by 60
  13. 2024Towards Lifelong Learning of Large Language Models: A Surveycited by 60
  14. 2023Scaling Expert Language Models with Unsupervised Domain Discoverycited by 59
  15. 2024Investigating Continual Pretraining in Large Language Models: Insights and Implicationscited by 54
  16. 2023Knowledge Card: Filling LLMs' Knowledge Gaps with Plug-in Specialized Language Modelscited by 46
  17. 2024Recent Advances of Foundation Language Models-based Continual Learning: A Surveycited by 46
  18. 2024D-CPT Law: Domain-specific Continual Pre-Training Scaling Law for Large Language Modelscited by 39
  19. 2024Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Modelscited by 38
  20. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  21. 2024Breaking the Curse of Multilinguality with Cross-lingual Expert Language Modelscited by 36
  22. 2023Mixture-of-Domain-Adapters: Decoupling and Injecting Domain Knowledge to Pre-trained Language Models Memoriescited by 34
  23. 2022Continual Training of Language Models for Few-Shot Learningcited by 33
  24. 2023How Do Large Language Models Capture the Ever-changing World Knowledge? A Review of Recent Advancescited by 33
  25. 2021Efficient Hierarchical Domain Adaptation for Pretrained Language Modelscited by 32
  26. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  27. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  28. 2024Model Swarms: Collaborative Search to Adapt LLM Experts via Swarm Intelligencecited by 27
  29. 2022M2D2: A Massively Multi-domain Language Modeling Datasetcited by 26
  30. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  31. 2023mmT5: Modular Multilingual Pre-Training Solves Source Language Hallucinationscited by 24
  32. 2024Towards Efficient Mixture of Experts: A Holistic Study of Compression Techniquescited by 24
  33. 2024Self-MoE: Towards Compositional Large Language Models with Self-Specialized Expertscited by 22
  34. 2024Unlocking Continual Learning Abilities in Language Modelscited by 22
  35. 2022KALA: Knowledge-Augmented Language Model Adaptationcited by 21
  36. 2024Layerwise Recurrent Router for Mixture-of-Expertscited by 21
  37. 2023ModuleFormer: Modularity Emerges from Mixture-of-Expertscited by 17
  38. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 16

50 most cited of 130 All 130 (JSON)