# Quantum-Like Contextuality in Large Language Models

arXiv: https://arxiv.org/abs/2412.16806 · HTML: https://mlsanity.com/paper/2412.16806/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 38 references that are in the map.
- First posted on arXiv 21 Dec 2024.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Neural Sheaf Diffusion: A Topological Perspective on Heterophily and Oversmoothing in GNNs](https://mlsanity.com/paper/2202.04579/) (2022)
- [An enriched category theory of language: from syntax to semantics](https://mlsanity.com/paper/2106.07890/) (2021)
- [Prospects for inconsistency detection using large language models and sheaves](https://mlsanity.com/paper/2401.16713/) (2024)
- [On the Quantum-like Contextuality of Ambiguous Phrases](https://mlsanity.com/paper/2107.14589/) (2021)
- [Sheaves as a Framework for Understanding and Interpreting Model Fit](https://mlsanity.com/paper/2105.10414/) (2021)
- [On Contextuality in Behavioral Data](https://mlsanity.com/paper/1508.04751/) (2015)
- [Generalised Winograd Schema and its Contextuality](https://mlsanity.com/paper/2308.16498/) (2023)
- [A Model of Anaphoric Ambiguities using Sheaf Theoretic Quantum-like Contextuality and BERT](https://mlsanity.com/paper/2208.05720/) (2022)
- [On monogamy of non-locality and macroscopic averages: examples and preliminary results](https://mlsanity.com/paper/1412.8541/) (2014)
- [Developments in Sheaf-Theoretic Models of Natural Language Ambiguities](https://mlsanity.com/paper/2402.04505/) (2024)

## Built on by (most cited first, 4 of 4)

- [The production of meaning in the processing of natural language](https://mlsanity.com/paper/2603.20381/) (2026)
- [Auditing Question-Order Effects in Large Language Models with the QQ Equality: Mechanism Characterization and a Saturation Caveat](https://mlsanity.com/paper/2607.17219/) (2026)
- [Algebraic Quantum Intelligence: A New Framework for Reproducible Machine Creativity](https://mlsanity.com/paper/2602.14130/) (2026)
- [Phase-Associative Memory: Sequence Modeling in Complex Hilbert Space](https://mlsanity.com/paper/2604.05030/) (2026)

## Abstract (arXiv)

Contextuality is a distinguishing feature of quantum mechanics and there is growing evidence that it is a necessary condition for quantum advantage. In order to make use of it, researchers have been asking whether similar phenomena arise in other domains. The answer has been yes, e.g. in behavioural sciences. However, one has to move to frameworks that take some degree of signalling into account. Two such frameworks exist: (1) a signalling-corrected sheaf theoretic model, and (2) the Contextuality-by-Default (CbD) framework. This paper provides the first large scale experimental evidence for a yes answer in natural language. We construct a linguistic schema modelled over a contextual quantum scenario, instantiate it in the Simple English Wikipedia and extract probability distributions for the instances using the large language model BERT. This led to the discovery of 77,118 sheaf-contextual and 36,938,948 CbD contextual instances. We proved that the contextual instances came from semantically similar words, by deriving an equation between degrees of contextuality and Euclidean distances of BERT's embedding vectors. A regression model further reveals that Euclidean distance is indeed the best statistical predictor of contextuality. Our linguistic schema is a variant of the co-reference resolution challenge. These results are an indication that quantum methods may be advantageous in language tasks.