# Overlapping Schwarz Attention: Hierarchical Attention via Domain Decomposition

arXiv: https://arxiv.org/abs/2606.18525 · HTML: https://mlsanity.com/paper/2606.18525/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 16 of its 30 references that are in the map.
- First posted on arXiv 16 Jun 2026.

## Builds on (most cited first, 16 of 16)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Fourier Neural Operator for Parametric Partial Differential Equations](https://mlsanity.com/paper/2010.08895/) (2020)
- [DeepONet: Learning nonlinear operators for identifying differential equations based on the universal approximation theorem of operators](https://mlsanity.com/paper/1910.03193/) (2019)
- [Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inference](https://mlsanity.com/paper/2406.10774/) (2024)
- [Choose a Transformer: Fourier or Galerkin](https://mlsanity.com/paper/2105.14995/) (2021)
- [An Exploration of Hierarchical Attention Transformers for Efficient Long Document Classification](https://mlsanity.com/paper/2210.05529/) (2022)
- [HANet: A Hierarchical Attention Network for Change Detection With Bitemporal Very-High-Resolution Remote Sensing Images](https://mlsanity.com/paper/2404.09178/) (2024)
- [Star Attention: Efficient LLM Inference over Long Sequences](https://mlsanity.com/paper/2411.17116/) (2024)
- [Finite Element Operator Network for Solving Elliptic-type parametric PDEs](https://mlsanity.com/paper/2308.04690/) (2023)
- [Rectified Sparse Attention](https://mlsanity.com/paper/2506.04108/) (2025)
- [HDT: Hierarchical Document Transformer](https://mlsanity.com/paper/2407.08330/) (2024)
- [Fine- and Coarse-Granularity Hybrid Self-Attention for Efficient BERT](https://mlsanity.com/paper/2203.09055/) (2022)
- [Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problems](https://mlsanity.com/paper/2509.15448/) (2025)

## Built on by (most cited first, 1 of 1)

- [How Token Influence Decays with Distance: A Green-Function View of Trained Language Models](https://mlsanity.com/paper/2606.29139/) (2026)

## Abstract (arXiv)

We propose a hierarchical attention mechanism based on two-level overlapping Schwarz domain decomposition. The method is motivated by domain decomposition methods in partial differential equations which combine local subdomain corrections with a coarse level that communicates global, long-range information. We test its usefulness in the context of finite-dimensional operator learning using a simple, one-dimensional diffusion problem. Although elementary, this problem provides a controlled sequence-to-sequence setting in which the exact nonlocal solution operator is known. After discretization, learning the solution operator amounts to approximating the inverse of a symmetric positive definite matrix. As a baseline, we use a global softmax-free low-rank attention operator of the form (QKT). The proposed construction replaces this global factorization by a two-level additive structure: local low-rank attention blocks on overlapping subdomains are combined with a coarse attention block. The resulting operator has the form  Mθ-1 = Phi Q0 K0T PhiT + sumi=1N RiT Di1/2 Qi KiT Di1/2 Ri .  Here, Ri restricts to an overlapping subdomain, Di is a partition-of-unity weight, and Phi is a coarse interpolation %(or prolongation) matrix. Numerical experiments for synthetic Fourier right-hand sides indicate that the domain-decomposition attention operator can converge faster and can give more accurate approximations than a global low-rank attention baseline while using significantly fewer parameters.