# QAMA: Scalable Quantum Annealing Multi-Head Attention Operator for Deep Learning

arXiv: https://arxiv.org/abs/2504.11083 · HTML: https://mlsanity.com/paper/2504.11083/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 53 references that are in the map.
- First posted on arXiv 15 Apr 2025.

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Generating Diverse High-Fidelity Images with VQ-VAE-2](https://mlsanity.com/paper/1906.00446/) (2019)
- [On The Computational Complexity of Self-Attention](https://mlsanity.com/paper/2209.04881/) (2022)
- [Better plain ViT baselines for ImageNet-1k](https://mlsanity.com/paper/2205.01580/) (2022)
- [Combiner: Full Attention Transformer with Sparse Computation Cost](https://mlsanity.com/paper/2107.05768/) (2021)
- [QSAN: A Near-term Achievable Quantum Self-Attention Network](https://mlsanity.com/paper/2207.07563/) (2022)
- [QKSAN: A Quantum Kernel Self-Attention Network](https://mlsanity.com/paper/2308.13422/) (2023)
- [HAQJSK: Hierarchical-Aligned Quantum Jensen-Shannon Kernels for Graph Classification](https://mlsanity.com/paper/2211.02904/) (2022)
- [Quantum Language Model with Entanglement Embedding for Question Answering](https://mlsanity.com/paper/2008.09943/) (2020)
- [PRformer: Pyramidal Recurrent Transformer for Multivariate Time Series Forecasting](https://mlsanity.com/paper/2408.10483/) (2024)
- [QuanTest: Entanglement-Guided Testing of Quantum Neural Network Systems](https://mlsanity.com/paper/2402.12950/) (2024)
- [Quantum Adjoint Convolutional Layers for Effective Data Representation](https://mlsanity.com/paper/2404.17378/) (2024)
- [Quantum Annealing based Power Grid Partitioning for Parallel Simulation](https://mlsanity.com/paper/2408.04097/) (2024)

## Built on by (most cited first, 2 of 2)

- [Boltzmann Attention: Learnable Ising Couplings for Cooperative Attention](https://mlsanity.com/paper/2606.12478/) (2026)
- [Variational-Ising-Attention:Tailored Attention Matters for Science](https://mlsanity.com/paper/2607.23634/) (2026)

## Abstract (arXiv)

Attention mechanisms underpin modern deep learning, while the quadratic time and space complexity limit scalability for long sequences. To address this, Quantum Annealing Multi-Head Attention (QAMA) is proposed, a novel drop-in operator that reformulates attention as an energy-based Hamiltonian optimization problem. In this framework, token interactions are encoded into binary quadratic terms, and quantum annealing is employed to search for low-energy configurations that correspond to effective attention patterns. Unlike classical sparse or approximate attention methods that rely on hand-crafted heuristics, QAMA allows sparsity structures to emerge naturally from the optimization process. Theoretically, computational complexity is analysed through single-spin flip dynamics, providing time to solution runtime bounds that depend on the spectral properties of the annealing Hamiltonian. Empirically, evaluation on both natural language and vision benchmarks shows that, across tasks, accuracy deviates by at most 2.7 points from standard multi-head attention, while requiring only linear qubits in sequence length. Visualizations further reveal that the Hamiltonian penalty terms induce meaningful and interpretable sparsity across heads. Finally, deployment on a coherent Ising machine validates the feasibility of running QAMA on real quantum hardware, showing tangible inference-time reductions compared with classical implementations. These results highlight QAMA as a pioneering and scalable step toward integrating quantum optimization devices into deep neural architectures, providing a seamlessly integrable and hardware-compatible alternative to conventional attention mechanisms. This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.