# AttentionX: Exploiting Consensus Discrepancy In Attention from A Distributed Optimization Perspective

arXiv: https://arxiv.org/abs/2409.04275 · HTML: https://mlsanity.com/paper/2409.04275/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 21 of its 27 references that are in the map.
- First posted on arXiv 6 Sep 2024.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [FedSplit: An algorithmic framework for fast federated optimization](https://mlsanity.com/paper/2005.05238/) (2020)
- [Exact Diffusion Inversion via Bi-directional Integration Approximation](https://mlsanity.com/paper/2307.10829/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [Distributed Optimization Using the Primal-Dual Method of Multipliers](https://mlsanity.com/paper/1702.00841/) (2017)
- [Privacy-Preserving Distributed Optimization via Subspace Perturbation: A General Framework](https://mlsanity.com/paper/2004.13999/) (2020)
- [BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences](https://mlsanity.com/paper/2403.09347/) (2024)
- [Derivation and Analysis of the Primal-Dual Method of Multipliers Based on Monotone Operator Theory](https://mlsanity.com/paper/1706.02654/) (2017)
- [Privacy-Preserving Distributed Processing: Metrics, Bounds, and Algorithms](https://mlsanity.com/paper/2009.01098/) (2020)
- [Adaptive Differentially Quantized Subspace Perturbation (ADQSP): A Unified Framework for Privacy-Preserving Distributed Average Consensus](https://mlsanity.com/paper/2312.07947/) (2023)
- [Distributed Optimisation with Linear Equality and Inequality Constraints using PDMM](https://mlsanity.com/paper/2309.12897/) (2023)
- [Revisiting the Primal-Dual Method of Multipliers for Optimisation over Centralised Networks](https://mlsanity.com/paper/2107.08809/) (2021)
- [On Suppressing Range of Adaptive Stepsizes of Adam to Improve Generalisation Performance](https://mlsanity.com/paper/2302.01029/) (2023)

## Abstract (arXiv)

In this paper, we extend the standard Attention in transformer by exploiting the consensus discrepancy from a distributed optimization perspective, referred to as AttentionX. It is noted that the primal-dual method of multipliers (PDMM) citeZhang16PDMM is designed to iteratively solve a broad class of distributed optimization problems over a pear-to-pear (P2P) network, where neighbouring nodes gradually reach consensus as specified by predefined linear edge-constraints in the optimization process. In particular, at each iteration of PDMM, each node in a network first performs information-gathering from neighbours and then performs local information-fusion. From a high-level point of view, the KQ-softmax-based weighted summation of V-representations in Attention corresponds information-gathering from neighbours while the feature-processing via the feed-forward network (FFN) in transformer corresponds to local information fusion. PDMM exploits the Lagrangian multipliers to capture the historical consensus discrepancy in the form of residual errors of the linear edge-constraints, which plays a crucial role for the algorithm to converge. Inspired by PDMM, we propose AttentionX to incorporate the consensus discrepancy in the output update-expression of the standard Attention. The consensus discrepancy in AttentionX refers to the difference between the weighted summation of V-representations and scaled V-representions themselves. Experiments on ViT and nanoGPT show promising performance.