mlsanityData to 14 Oct 2026

Multi-Head Attention with Disagreement Regularization

arXiv 1810.10183 ↗First posted 24 Oct 2018cs.CL cs.AIcited by 35

Not on the map. Why?

Abstract (arXiv)

Multi-head attention is appealing for the ability to jointly attend to information from different representation subspaces at different positions. In this work, we introduce a disagreement regularization to explicitly encourage the diversity among multiple attention heads. Specifically, we propose three types of disagreement regularization, which respectively encourage the subspace, the attended positions, and the output representation associated with each attention head to be different from other heads. Experimental results on widely-used WMT14 English-German and WMT17 Chinese-English translation tasks demonstrate the effectiveness and universality of the proposed approach.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Oct 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 8; 2022: 9; 2023: 7; 2024: 2; 2025: 3; 2026: 5.

35 papers.

Sort:

By area:

  1. 2021A Survey of Transformerscited by 401
  2. 2021Universal Simultaneous Machine Translation with Mixture-of-Experts Wait-k Policycited by 32
  3. 2023DealMVC: Dual Contrastive Calibration for Multi-view Clusteringcited by 19
  4. 2022BeautyREC: Robust, Efficient, and Content-preserving Makeup Transfercited by 16
  5. 2021Alignment Attention by Matching Key and Query Distributionscited by 14
  6. 2024Explaining Modern Gated-Linear RNNs via a Unified Implicit Attention Formulationcited by 11
  7. 2024LookHere: Vision Transformers with Directed Attention Generalize and Extrapolatecited by 10
  8. 2022Leveraging Smooth Attention Prior for Multi-Agent Trajectory Predictioncited by 9
  9. 2023ChatGPT-Like Large-Scale Foundation Models for Prognostics and Health Management: A Survey and Roadmapscited by 8
  10. 2020Adversarial Mixture Of Experts with Category Hierarchy Soft Constraintcited by 6
  11. 2022Paying More Attention to Self-attention: Improving Pre-trained Language Models via Attention Guidingcited by 6
  12. 2021On Biasing Transformer Attention Towards Monotonicitycited by 5
23 more
  1. 2022Relaxed Attention for Transformer Modelscited by 5
  2. 2022Analysis of Self-Attention Head Diversity for Conformer-based Automatic Speech Recognitioncited by 3
  3. 2026TensorLens: End-to-End Transformer Analysis via High-Order Attention Tensorscited by 3
  4. 2026Order Matters in Retrosynthesis: Structure-aware Generation via Reaction-Center-Guided Discrete Flow Matchingcited by 3
  5. 2021Maximizing Mutual Information Across Feature and Topology Views for Learning Graph Representationscited by 2
  6. 2021Discrete Auto-regressive Variational Attention Models for Text Modelingcited by 2
  7. 2022Influential Recommender Systemcited by 2
  8. 2025Neural Diversity Regularizes Hallucinations in Language Modelscited by 2
  9. 2026Multi-Head Attention Is a Multi-Player Gamecited by 2
  10. 2022Horizontal and Vertical Attention in Transformerscited by 1
  11. 2022Explanation on Pretraining Bias of Finetuned Vision Transformercited by 1
  12. 2022EIT: Enhanced Interactive Transformercited by 1
  13. 2023Finding the Pillars of Strength for Multi-Head Attentioncited by 1
  14. 2023Multi-Objective Decision Transformers for Offline Reinforcement Learningcited by 1
  15. 2023PartialFormer: Modeling Part Instead of Whole for Machine Translationcited by 1
  16. 2025Patch Pruning Strategy Based on Robust Statistical Measures of Attention Weight Diversity in Vision Transformerscited by 1
  17. 2026Contradiction to Consensus: Dual Perspective, Multi Source Retrieval Based Claim Verification with Source Level Disagreement using LLMcited by 1
  18. 2021Cascaded Head-colliding Attentioncited by 0
  19. 2021Learning Slice-Aware Representations with Mixture of Attentionscited by 0
  20. 2023Towards Reliable Neural Machine Translation with Consistency-Aware Meta-Learningcited by 0
  21. 2023Disentangling the Linguistic Competence of Privacy-Preserving BERTcited by 0
  22. 2025Style4Rec: Enhancing Transformer-based E-commerce Recommendation Systems with Style and Shopping Cart Informationcited by 0
  23. 2026Transformer-Enhanced Reinforcement Learning: Fundamentals and Applications in Communication Networkscited by 0