mlsanityData to 7 Oct 2026

LLM-Based Test Oracles: Source-of-Authority Taxonomy -- A Systematic Literature Review

arXiv 2607.05031 ↗First posted 6 Jul 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) increasingly decide whether software behaves correctly, either by writing a test oracle or by acting as one. Yet two oracles can look identical and rest on different ground: one assertion encodes a written specification, another only what the model learned in training. Prior secondary studies sort oracles by form or by technique, rarely by the property that governs how far a verdict can be trusted: where its authority comes from. This systematic literature review, reported under the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines, screens 2,436 records to 54 included studies, extended by citation searching (snowballing) to 83 in total. We read the corpus along three axes: the source of an oracle's authority, the form it takes, and the mechanism that adjudicates it. Just over half of the corpus reaches a verdict with no specification at all. That is what lets these oracles work on code with no specification to consult, and what leaves a challenged verdict with less to fall back on. Source and mechanism cross-cut rather than coincide, so a label such as LLM-as-a-judge names how a verdict is produced, not why it should be trusted. Oracle quality is most often judged by resemblance to a known oracle rather than by whether injected faults are caught. The first question to ask of any LLM oracle is therefore what one would point to in defending its verdict. The protocol, search query, and per-study coding sheet are released.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 16 of its 82 references that are in the map. First posted on arXiv 6 Jul 2026.

Builds on

16 of 82 references.

References · 16

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2016Variational Inference: A Review for Statisticianscited by 1,466
  3. 2018On the Spectral Bias of Neural Networkscited by 1,321
  4. 2019Normalizing Flows for Probabilistic Modeling and Inferencecited by 1,315
  5. 2018Accurate Uncertainties for Deep Learning Using Calibrated Regressioncited by 417
  6. 2016Tutorial on Variational Autoencoderscited by 404
  7. 2018How to Start Training: The Effect of Initialization and Architecturecited by 109
  8. 2020A Review of Vibration-Based Damage Detection in Civil Structures: From Traditional Methods to Machine Learning and Deep Learning Applicationscited by 33
  9. 2019Solving Bayesian Inverse Problems via Variational Autoencoderscited by 32
  10. 2021Value of information from vibration-based structural health monitoring extracted via Bayesian model updatingcited by 10
  11. 2020Error Control and Loss Functions for the Deep Learning Inversion of Borehole Resistivity Measurementscited by 8
  12. 2024Response Estimation and System Identification of Dynamical Systems via Physics-Informed Neural Networkscited by 4
4 more
  1. 2024Stabilizing the Kumaraswamy Distributioncited by 3
  2. 2025PINNverse: Accurate parameter estimation in differential equations from noisy data with constrained physics-informed neural networkscited by 3
  3. 2024Scientific Machine Learning Seismologycited by 2
  4. 2025A Copula-based variational autoencoder for uncertainty quantification in inverse problems: application to damage identification in an offshore wind turbinecited by 1

Built on by

Not cited yet.