mlsanityData to 7 Oct 2026

Are queries and keys always relevant? A case study on Transformer wave functions

arXiv 2405.18874 ↗First posted 29 May 2024cond-mat.dis-nn cs.CL physics.comp-phcited by 5

Not on the map. Why?

Abstract (arXiv)

The dot product attention mechanism, originally designed for natural language processing tasks, is a cornerstone of modern Transformers. It adeptly captures semantic relationships between word pairs in sentences by computing a similarity overlap between queries and keys. In this work, we explore the suitability of Transformers, focusing on their attention mechanisms, in the specific domain of the parametrization of variational wave functions to approximate ground states of quantum many-body spin Hamiltonians. Specifically, we perform numerical simulations on the two-dimensional J1-J2 Heisenberg model, a common benchmark in the field of quantum many-body systems on lattice. By comparing the performance of standard attention mechanisms with a simplified version that excludes queries and keys, relying solely on positions, we achieve competitive results while reducing computational cost and parameter usage. Furthermore, through the analysis of the attention maps generated by standard attention mechanisms, we show that the attention weights become effectively input-independent at the end of the optimization. We support the numerical results with analytical calculations, providing physical insights of why queries and keys should be, in principle, omitted from the attention mechanism when studying large systems.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 25 of its 66 references that are in the map. First posted on arXiv 29 May 2024.

Builds on

25 of 66 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2018Self-Attention with Relative Position Representationscited by 1,112
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2020On Layer Normalization in the Transformer Architecturecited by 906
  9. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  10. 2021CoAtNet: Marrying Convolution and Attention for All Data Sizescited by 566
  11. 2020Rethinking Positional Encoding in Language Pre-trainingcited by 197
  12. 2019Ab-Initio Solution of the Many-Electron Schr"odinger Equation with Deep Neural Networkscited by 195
13 more
  1. 2022Vision Transformers provably learn spatial structurecited by 92
  2. 2022A Self-Attention Ansatz for Ab-initio Quantum Chemistrycited by 45
  3. 2017Neural-Network Quantum States, String-Bond States, and Chiral Topological Statescited by 31
  4. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  5. 2018Fisher Information and Natural Gradient Learning of Random Deep Networkscited by 28
  6. 2021NetKet 3: Machine Learning Toolbox for Many-Body Quantum Systemscited by 27
  7. 2020Do Transformers Need Deep Long-Range Memorycited by 24
  8. 2021The Case for Translation-Invariant Self-Attention in Transformer-Based Language Modelscited by 23
  9. 2022The NLP Task Effectiveness of Long-Range Transformerscited by 22
  10. 2023Mapping of attention mechanisms to a generalized Potts modelcited by 19
  11. 2021Gauge Invariant and Anyonic Symmetric Transformer and RNN Quantum States for Quantum Lattice Modelscited by 17
  12. 2019Geometry of learning neural quantum statescited by 14
  13. 2024Ab-initio variational wave functions for the time-dependent many-electron Schr"odinger equationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Hyperbolic recurrent neural network as the first type of non-Euclidean neural quantum state ansatzcited by 3
  2. 2026Scalable Physics-Inspired Transformers for Spin Glassescited by 2
  3. 2025Explorative Curriculum Learning for Strongly Correlated Electron Systemscited by 1
  4. 2026New non-Euclidean neural quantum states from hyperbolic Lorentz recurrent architecturescited by 1
  5. 2025Fermionic neural Gibbs statescited by 0