# Improved Belief-Attention in Vision Task

arXiv: https://arxiv.org/abs/2606.00077 · HTML: https://mlsanity.com/paper/2606.00077/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 14 references that are in the map.
- First posted on arXiv 22 May 2026.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Scalable Diffusion Models with Transformers](https://mlsanity.com/paper/2212.09748/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioning](https://mlsanity.com/paper/2307.08691/) (2023)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Ring Attention with Blockwise Transformers for Near-Infinite Context](https://mlsanity.com/paper/2310.01889/) (2023)
- [Exact Diffusion Inversion via Bi-directional Integration Approximation](https://mlsanity.com/paper/2307.10829/) (2023)
- [Transformers in Speech Processing: A Survey](https://mlsanity.com/paper/2303.11607/) (2023)
- [BurstAttention: An Efficient Distributed Attention Framework for Extremely Long Sequences](https://mlsanity.com/paper/2403.09347/) (2024)
- [Exclusive Self Attention](https://mlsanity.com/paper/2603.09078/) (2026)

## Abstract (arXiv)

Recently, Belief-Attention citeGuoqiang25BeliefAttention has been proposed by first performing an orthogonal projection of the softmax-based weighted summation of V vectors with respect to the original V vectors and then taking the perpendicular component as the residual signal in Transformer for performance improvement. In this paper, we first conduct an ablation study showing the projected component also carries information about the token correlation, which should not be ignored. We then propose to extend Belief-Attention by making use of both the perpendicular and projected components. In particular, the projected component goes through certain activation function and then a linear mapping before merging with the considered token. Conceptually speaking, the neural block for the projected component can be viewed as a two-layer feedforward network (FFN) within the new attention block. It is also noted that standard attention captures the token correlation via the inner-product matrix QKT. We propose to introduce an additional inner-product matrix ZZT to QKT to capture richer token correlation. We refer to the new module as Belief2-Attention. It can be easily shown that Belief2-Attention is more expressive than standard Attention. We then verify the effectiveness of Belief2-Attention for vision tasks of image classification and segmentation.