mlsanityData to 7 Oct 2026

Transformers with Competitive Ensembles of Independent Mechanisms

arXiv 2103.00336 ↗First posted 27 Feb 2021cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

An important development in deep learning from the earliest MLPs has been a move towards architectures with structural inductive biases which enable the model to keep distinct sources of information and routes of processing well-separated. This structure is linked to the notion of independent mechanisms from the causality literature, in which a mechanism is able to retain the same processing as irrelevant aspects of the world are changed. For example, convnets enable separation over positions, while attention-based architectures (especially Transformers) learn which combination of positions to process dynamically. In this work we explore a way in which the Transformer architecture is deficient: it represents each position with a large monolithic hidden representation and a single set of parameters which are applied over the entire hidden representation. This potentially throws unrelated sources of information together, and limits the Transformer's ability to capture independent mechanisms. To address this, we propose Transformers with Independent Mechanisms (TIM), a new Transformer layer which divides the hidden representation and parameters into multiple mechanisms, which only exchange information through attention. Additionally, we propose a competition mechanism which encourages these mechanisms to specialize over time steps, and thus be more independent. We study TIM on a large-scale BERT model, on the Image Transformer, and on speech enhancement and find evidence for semantically meaningful specialization as well as improved performance.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 39 of its 64 references that are in the map. First posted on arXiv 27 Feb 2021.

Builds on

39 of 64 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2020Scaling Laws for Neural Language Modelscited by 6,411
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  8. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
27 more
  1. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  2. 2018Universal Transformerscited by 613
  3. 2015Neural Module Networkscited by 375
  4. 2017PathNet: Evolution Channels Gradient Descent in Super Neural Networkscited by 362
  5. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 273
  6. 2019Recurrent Independent Mechanismscited by 219
  7. 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 213
  8. 2019CATER: A diagnostic dataset for Compositional Actions and TEmporal Reasoningcited by 137
  9. 2020DCCRN: Deep Complex Convolution Recurrent Network for Phase-Aware Speech Enhancementcited by 133
  10. 2020The INTERSPEECH 2020 Deep Noise Suppression Challenge: Datasets, Subjective Testing Framework, and Challenge Resultscited by 128
  11. 2017Routing Networks: Adaptive Selection of Non-linear Functions for Multi-Task Learningcited by 125
  12. 2017Learning Independent Causal Mechanismscited by 104
  13. 2015Neural Programmer-Interpreterscited by 96
  14. 2016Disentangling factors of variation in deep representations using adversarial trainingcited by 87
  15. 2020BERTology Meets Biology: Interpreting Attention in Protein Language Modelscited by 85
  16. 2018Modular meta-learningcited by 51
  17. 2020Object Files and Schemata: Factorizing Declarative and Procedural Knowledge in Dynamical Systemscited by 48
  18. 2020Learning to Combine Top-Down and Bottom-Up Signals in Recurrent Neural Networks with Attention over Modulescited by 38
  19. 2019T-GSA: Transformer with Gaussian-weighted self-attention for speech enhancementcited by 34
  20. 2020PoCoNet: Better Speech Enhancement with Frequency-Positional Embeddings, Semi-Supervised Conversational Data, and Biased Losscited by 23
  21. 2020Repulsive Attention: Rethinking Multi-head Attention as Bayesian Inferencecited by 13
  22. 2020Exploring the Best Loss Function for DNN-Based Low-latency Speech Enhancement with Temporal Convolutional Networkscited by 10
  23. 2020S2RMs: Spatially Structured Recurrent Modulescited by 10
  24. 2020Phase-aware Single-stage Speech Denoising and Dereverberation with U-Netcited by 7
  25. 2017Contaminated speech training methods for robust DNN-HMM distant speech recognitioncited by 5
  26. 2020Boosting Objective Scores of a Speech Enhancement Model by MetricGAN Post-processingcited by 3
  27. 2020Neural Function Modules with Sparse Arguments: A Dynamic Approach to Integrating Information across Layerscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 5; 2023: 1; 2024: 5; 2025: 0; 2026: 1.

19 papers.

Sort:

By area:

  1. 2021Illiterate DALL-E Learns to Composecited by 147
  2. 2022Simple Unsupervised Object-Centric Learning for Complex and Naturalistic Videoscited by 139
  3. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  4. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  5. 2022Improving Object-centric Learning with Query Optimizationcited by 60
  6. 2021Discrete-Valued Neural Communicationcited by 39
  7. 2022Neural Systematic Bindercited by 37
  8. 2021Dynamic Inference with Neural Interpreterscited by 24
  9. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  10. 2024Interaction Asymmetry: A General Principle for Learning Composable Abstractionscited by 19
  11. 2021Compositional Attention: Disentangling Search and Retrievalcited by 12
  12. 2024SlotLifter: Slot-guided Feature Lifting for Learning Object-centric Radiance Fieldscited by 12
7 more
  1. 2022Adaptive Discrete Communication Bottlenecks with Dynamic Vector Quantizationcited by 11
  2. 2021Neural Attention Models in Deep Learning: Survey and Taxonomycited by 8
  3. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  4. 2023Discrete Messages Improve Communication Efficiency among Isolated Intelligent Agentscited by 1
  5. 2022Syntactic Inductive Biases for Deep Learning Methodscited by 0
  6. 2024Block-Operations: Using Modular Routing to Improve Compositional Generalizationcited by 0
  7. 2026Differentiable Learning of Lifted Action Schemas for Classical Planningcited by 0