mlsanityData to 7 Oct 2026

Quantum Transformer: Accelerating model inference via quantum linear algebra

arXiv 2402.16714 ↗First posted 26 Feb 2024quant-ph cs.AI cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

Powerful generative artificial intelligence from large language models (LLMs) harnesses extensive computational resources for inference. In this work, we investigate the transformer architecture, a key component of these models, under the lens of fault-tolerant quantum computing. We develop quantum subroutines to construct the building blocks in the transformer, including the self-attention, residual connection with layer normalization, and feed-forward network. As an important subroutine, we show how to efficiently implement the Hadamard product and element-wise functions of matrices on quantum computers. Our algorithm prepares an amplitude encoding of the transformer output, which can be measured for prediction or use in the next layer. We find that the matrix norm of the input sequence plays a dominant role in the quantum complexity. With numerical experiments on open-source LLMs, including for bio-informatics applications, we demonstrate the potential of a quantum speedup for transformer inference in practical regimes.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 41 of its 93 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

41 of 93 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2023GPT-4 Technical Reportcited by 18,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  8. 2016Layer Normalizationcited by 5,579
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  11. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  12. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
29 more
  1. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  2. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  3. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
  4. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  5. 2018Self-Attention Generative Adversarial Networkscited by 809
  6. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  7. 2020Efficient Transformers: A Surveycited by 784
  8. 2020Variational Quantum Algorithmscited by 687
  9. 2018Barren plateaus in quantum neural network training landscapescited by 624
  10. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  11. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  12. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  13. 2021SPLADE: Sparse Lexical and Expansion Model for First Stage Rankingcited by 283
  14. 2022Tensor Programs V: Tuning Large Neural Networks via Zero-Shot Hyperparameter Transfercited by 272
  15. 2020Predicting Many Properties of a Quantum System from Very Few Measurementscited by 229
  16. 2023HyenaDNA: Long-Range Genomic Sequence Modeling at Single Nucleotide Resolutioncited by 187
  17. 2021Between words and characters: A Brief History of Open-Vocabulary Modeling and Tokenization in NLPcited by 130
  18. 2021SPLADE v2: Sparse Lexical and Expansion Model for Information Retrievalcited by 126
  19. 2023Does provable absence of barren plateaus imply classical simulability?cited by 78
  20. 2018A quantum-inspired classical algorithm for recommendation systemscited by 70
  21. 2016Quantum Recommendation Systemscited by 69
  22. 2019Sampling-based sublinear low-rank matrix arithmetic framework for dequantizing quantum machine learningcited by 37
  23. 2018Quantum principal component analysis only achieves an exponential speedup because of its state preparation assumptionscited by 34
  24. 2023Towards provably efficient quantum algorithms for large-scale machine-learning modelscited by 32
  25. 2018The power of block-encoded matrix powers: improved regression techniques via faster Hamiltonian simulationcited by 30
  26. 2022Great Power, Great Responsibility: Recommendations for Reducing Energy for Training Language Modelscited by 21
  27. 2018Quantum-inspired low-rank stochastic regression with logarithmic dependence on the dimensioncited by 18
  28. 2021Nonlinear transformation of complex amplitudes via quantum singular value transformationcited by 9
  29. 2020An improved quantum-inspired algorithm for linear regressioncited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 5; 2026: 1.

8 papers.

  1. 2024Quantum Attention for Vision Transformers in High Energy Physicscited by 1
  2. 2024Learning Complex Word Embeddings in Classical and Quantum Spacescited by 1
  3. 2025Quantum Graph Attention Networks: Trainable Quantum Encoders for Inductive Graph Learningcited by 1
  4. 2025Accelerating Inference for Multilayer Neural Networks with Quantum Computerscited by 1
  5. 2025Quantum Complex-Valued Self-Attention Modelcited by 0
  6. 2025Efficient Generation of Parameterised Quantum Circuits from Large Textscited by 0
  7. 2025QFFN-BERT: An Empirical Study of Depth, Performance, and Data Efficiency in Hybrid Quantum-Classical Transformerscited by 0
  8. 2026Variational Quantum Transformer Architecture for Synthetic Language Generationcited by 0