mlsanityData to 14 Oct 2026

Auditing Question-Order Effects in Large Language Models with the QQ Equality: Mechanism Characterization and a Saturation Caveat

arXiv 2607.17219 ↗First posted 19 Jul 2026cs.CL cs.AI quant-phcited by 1

Not on the map. Why?

Abstract (arXiv)

Question-order effects in human survey data have been reported to approximately satisfy the QQ (quantum question) equality, a parameter-free prediction of the standard projective quantum question-order model. We develop this equality into an audit framework for sequential binary judgments of autoregressive large language models (LLMs). Theoretically, we characterize mechanism families that satisfy QQ robustly, show that classical repetition can reproduce the equality exactly, and combine QQ with the rank-2 Contextuality-by-Default criterion through |qQQ| ≤ OSS. This separates order sensitivity, QQ imbalance, and residual contextuality rather than treating them as interchangeable signatures. Methodologically, we introduce a committed multi-turn forced-branch protocol that reconstructs order-conditioned joint distributions from next-token log-probabilities under counterbalanced label mappings and pre-specified health gates. A first-signal pilot on an open-weight instruction-tuned model reveals the central measurement problem. Although all pre-specified health gates passed, the binary-conditioned distributions were near-deterministic for 17 of 18 item pairs under the direct-evaluation framing and 7 of 8 under the persona framing. Label assignment materially changed several mapping-specific QQ verdicts, and no item was certified as residually contextual. Thus, under the tested conditions, the observed QQ outcomes did not uniquely identify a response mechanism in the presence of a saturated and label-sensitive measurement interface. The main implication is methodological: next-token probabilities should not be interpreted as survey-response distributions without first establishing adequate dispersion. We therefore argue that saturation screening and label counterbalancing should precede structural interpretation in distribution-level audits of LLM judgments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 30 references that are in the map. First posted on arXiv 19 Jul 2026.

Builds on

14 of 30 references.

References · 14

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  3. 2023Large Language Models Are Not Robust Multiple Choice Selectorscited by 486
  4. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 396
  5. 2023Large Language Models Sensitivity to The Order of Options in Multiple-Choice Questionscited by 263
  6. 2023Does Writing with Language Models Reduce Content Diversity?cited by 192
  7. 2023Do LLMs exhibit human-like response biases? A case study in survey designcited by 180
  8. 2024"My Answer is C": First-Token Probabilities Do Not Match Text Answers in Instruction-Tuned Language Modelscited by 69
  9. 2025Prompt Perturbations Reveal Human-Like Biases in Large Language Model Survey Responsescited by 13
  10. 2024Quantum-Like Contextuality in Large Language Modelscited by 4
  11. 2025CAPE: Context-Aware Personality Evaluation Framework for Large Language Modelscited by 4
  12. 2025Identifying Quantum Structure in AI Language: Evidence for Evolutionary Convergence of Human and Artificial Cognitioncited by 3
2 more
  1. 2026Failure of contextual invariance in large language modelscited by 3
  2. 2026Ontology-Amplified Distillation and Contextuality Auditing for Sovereign Enterprise Language Models: A Combined Proof-of-Mechanism and Negative-Results Method Studycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.