# Multi-Head Attention with Disagreement Regularization

arXiv: https://arxiv.org/abs/1810.10183 · HTML: https://mlsanity.com/paper/1810.10183/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Oct 2018.

## Built on by (most cited first, 30 of 35)

- [A Survey of Transformers](https://mlsanity.com/paper/2106.04554/) (2021)
- [Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policy](https://mlsanity.com/paper/2109.05238/) (2021)
- [DealMVC: Dual Contrastive Calibration for Multi-view Clustering](https://mlsanity.com/paper/2308.09000/) (2023)
- [BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfer](https://mlsanity.com/paper/2212.05855/) (2022)
- [Alignment Attention by Matching Key and Query Distributions](https://mlsanity.com/paper/2110.12567/) (2021)
- [Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulation](https://mlsanity.com/paper/2405.16504/) (2024)
- [LookHere: Vision Transformers with Directed Attention Generalize and Extrapolate](https://mlsanity.com/paper/2405.13985/) (2024)
- [Leveraging Smooth Attention Prior for Multi-Agent Trajectory Prediction](https://mlsanity.com/paper/2203.04421/) (2022)
- [ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmaps](https://mlsanity.com/paper/2305.06472/) (2023)
- [Adversarial Mixture Of Experts with Category Hierarchy Soft Constraint](https://mlsanity.com/paper/2007.12349/) (2020)
- [Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guiding](https://mlsanity.com/paper/2204.02922/) (2022)
- [On Biasing Transformer Attention Towards Monotonicity](https://mlsanity.com/paper/2104.03945/) (2021)
- [Relaxed Attention for Transformer Models](https://mlsanity.com/paper/2209.09735/) (2022)
- [Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognition](https://mlsanity.com/paper/2209.06096/) (2022)
- [TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensors](https://mlsanity.com/paper/2601.17958/) (2026)
- [Order Matters in Retrosynthesis: Structure-aware Generation via Reaction-Center-Guided Discrete Flow Matching](https://mlsanity.com/paper/2602.13136/) (2026)
- [Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representations](https://mlsanity.com/paper/2105.06715/) (2021)
- [Discrete Auto-regressive Variational Attention Models for Text Modeling](https://mlsanity.com/paper/2106.08571/) (2021)
- [Influential Recommender System](https://mlsanity.com/paper/2211.10002/) (2022)
- [Neural Diversity Regularizes Hallucinations in Language Models](https://mlsanity.com/paper/2510.20690/) (2025)
- [Multi-Head Attention Is a Multi-Player Game](https://mlsanity.com/paper/2602.00861/) (2026)
- [Horizontal and Vertical Attention in Transformers](https://mlsanity.com/paper/2207.04399/) (2022)
- [Explanation on Pretraining Bias of Finetuned Vision Transformer](https://mlsanity.com/paper/2211.15428/) (2022)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [Finding the Pillars of Strength for Multi-Head Attention](https://mlsanity.com/paper/2305.14380/) (2023)
- [Multi-Objective Decision Transformers for Offline Reinforcement Learning](https://mlsanity.com/paper/2308.16379/) (2023)
- [PartialFormer: Modeling Part Instead of Whole for Machine Translation](https://mlsanity.com/paper/2310.14921/) (2023)
- [Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformers](https://mlsanity.com/paper/2507.19175/) (2025)
- [Contradiction to Consensus: Dual Perspective, Multi Source Retrieval Based Claim Verification with Source Level Disagreement using LLM](https://mlsanity.com/paper/2602.18693/) (2026)
- [Cascaded Head-colliding Attention](https://mlsanity.com/paper/2105.14850/) (2021)

## Abstract (arXiv)

Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.