mlsanityData to 14 Oct 2026

Emergent Modularity in Pre-trained Transformers

arXiv 2305.18390 ↗First posted 28 May 2023cs.CL cs.LGcited by 37

Not on the map. Why?

Abstract (arXiv)

This work examines the presence of modularity in pre-trained Transformers, a feature commonly found in human brains and thought to be vital for general intelligence. In analogy to human brains, we consider two main characteristics of modularity: (1) functional specialization of neurons: we evaluate whether each neuron is mainly specialized in a certain function, and find that the answer is yes. (2) function-based neuron grouping: we explore finding a structure that groups neurons into modules by function, and each module works for its corresponding function. Given the enormous amount of possible structures, we focus on Mixture-of-Experts as a promising candidate, which partitions neurons into experts and usually activates different experts for different inputs. Experimental results show that there are functional experts, where clustered are the neurons specialized in a certain function. Moreover, perturbing the activations of functional experts significantly affects the corresponding function. Finally, we study how modularity emerges during pre-training, and find that the modular structure is stabilized at the early stage, which is faster than neuron stabilization. It suggests that Transformers first construct the modular structure and then learn fine-grained neuron functions. Our code and data are available at https://github.com/THUNLP/modularity-analysis.

Key facts. Cited by 37 papers in the map (data through 14 Oct 2026). Builds on 50 of its 83 references that are in the map. First posted on arXiv 28 May 2023.

Builds on

50 of 83 references.

References · 50

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  12. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
38 more
  1. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  2. 2019Language Models as Knowledge Bases?cited by 1,936
  3. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  7. 2019How Can We Know What Language Models Know?cited by 959
  8. 2018Neural Network Acceptability Judgmentscited by 958
  9. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  10. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  11. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  12. 2022A Generalist Agentcited by 808
  13. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  14. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  15. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  16. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 511
  17. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  18. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  19. 2015Neural Module Networkscited by 376
  20. 2021Measuring and Improving Consistency in Pretrained Language Modelscited by 364
  21. 2021BASE Layers: Simplifying Training of Large, Sparse Modelscited by 320
  22. 2019Recurrent Independent Mechanismscited by 219
  23. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  24. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 152
  25. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  26. 2022Lifting the Curse of Multilinguality by Pre-training Modular Transformerscited by 146
  27. 2021DEMix Layers: Disentangling Domains for Modular Language Modelingcited by 132
  28. 2019Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)cited by 128
  29. 2021DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learningcited by 115
  30. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  31. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 98
  32. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  33. 2022Finding Skill Neurons in Pre-trained Transformer-based Language Modelscited by 66
  34. 2018Modular Networks: Learning to Decompose Neural Computationcited by 62
  35. 2021Disentangling Syntax and Semantics in the Brain with Deep Networkscited by 40
  36. 2020Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modulescited by 38
  37. 2021How transfer learning impacts linguistic knowledge in deep NLP models?cited by 32
  38. 2020Finding Experts in Transformer Modelscited by 28

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 12; 2025: 10; 2026: 10.

37 papers.

Sort:

By area:

  1. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 67
  2. 2024ConceptPrune: Concept Editing in Diffusion Models via Skilled Neuron Pruningcited by 55
  3. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  4. 2024Sparsing Law: Towards Large Language Models with Greater Activation Sparsitycited by 22
  5. 2024Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Designcited by 19
  6. 2023Exploiting Activation Sparsity with Dense to Dynamic-k Mixture-of-Experts Conversioncited by 16
  7. 2025Towards Enhanced Image Generation Via Multi-modal Chain of Thought in Unified Generative Modelscited by 14
  8. 2023Examining Modularity in Multilingual LMs via Language-Specialized Subnetworkscited by 13
  9. 2024Memorized Images in Diffusion Models share a Subspace that can be Located and Deletedcited by 13
  10. 2024Conditional computation in neural networks: principles and research trendscited by 10
  11. 2025Mixture of Cognitive Reasoners: Modular Reasoning with Brain-Like Specializationcited by 9
  12. 2025From Neurons to Semantics: Evaluating Cross-Linguistic Alignment Capabilities of Large Language Models via Neurons Alignmentcited by 9
25 more
  1. 2023Customizable Combination of Parameter-Efficient Modules for Multi-Task Learningcited by 8
  2. 2024ELICIT: LLM Augmentation via External In-Context Capabilitycited by 7
  3. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 7
  4. 2026Position: Weight Space Should Be a First-Class Generative AI Modalitycited by 6
  5. 2024Exploring the Benefit of Activation Sparsity in Pre-trainingcited by 5
  6. 2024Neuralink: Fast LLM Inference on Smartphones with Neuron Co-Activation Linkingcited by 5
  7. 2025Variance-Based Pruning for Accelerating and Compressing Trained Networkscited by 5
  8. 2024Investigating Neuron Ablation in Attention Heads: The Case for Peak Activation Centeringcited by 4
  9. 2025Efficient Data Driven Mixture-of-Expert Extraction from Trained Networkscited by 3
  10. 2025Continual Personalization for Diffusion Modelscited by 3
  11. 2026Component-Level Lesioning of Language Models Reveals Clinically Aligned Aphasia Phenotypescited by 3
  12. 2023Assessing the nature of large language models: A caution against anthropocentrismcited by 2
  13. 2023Advancing Perception in Artificial Intelligence through Principles of Cognitive Sciencecited by 1
  14. 2025Federated Fine-Tuning of Sparsely-Activated Large Language Models on Resource-Constrained Devicescited by 1
  15. 2025Neuronal Group Communication for Efficient Neural representationcited by 1
  16. 2025Beyond Tokens in Language Models: Interpreting Activations through Text Genre Chunkscited by 1
  17. 2026GPrune-LLM: Generalization-Aware Structured Pruning for Large Language Modelscited by 1
  18. 2024Modularity in Transformers: Investigating Neuron Separability & Specializationcited by 0
  19. 2026Target Concept Tuning Improves Extreme Weather Forecastingcited by 0
  20. 2026Structural Graph Probing of Vision-Language Modelscited by 0
  21. 2026Modular Representation Compression: Adapting LLMs for Efficient and Effective Recommendationscited by 0
  22. 2026Learning Emergent Modular Representations in Multi-modality Medical Vision Foundation Modelscited by 0
  23. 2026Modularity-Free Conflict-Averse Training for Generalized PINNscited by 0
  24. 2026Affinity-Aware Sharding for Delayed Tensor Parallelismcited by 0
  25. 2026CORTEX: Learning to Share and Specialize in Dense Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Modular Cognitive Architecture Emerges in Large Language Models
  2. 2025Modular connectivity in neural networks emerges from Poisson noise-motivated regularisation, and promotes robustness and compositional generalisation
  3. 2026Neural networks are more modular than single neurons suggest
  4. 2024Configurable Foundation Models: Building LLMs from a Modular Perspective
  5. 2023Unlocking Emergent Modularity in Large Language Models
  6. 2026Pre-carved Niches: The Formation Dynamics of Modular Task Partitions in Early LLM Training
  7. 2024Fourier Circuits in Neural Networks and Transformers: A Case Study of Modular Arithmetic with Multiple Inputs
  8. 2023Seeing is Believing: Brain-Inspired Modular Training for Mechanistic Interpretability