mlsanityData to 14 Oct 2026

Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphs

arXiv 2607.12273 ↗First posted 14 Jul 2026cs.SE cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

As Code Large Language Models (LLMs) become central to modern software engineering, their inherent stochasticity poses significant real-world risks, where even minor errors can lead to severe functional, security, or safety consequences. Reliable automation, therefore, demands the ability to distinguish between confident, well-supported predictions and stochastic guessing. However, existing uncertainty estimation methods face a critical gap: white and grey-box techniques are often inapplicable to closed-source models, while standard "black-box" text metrics fail to capture the unique fragility of code, where syntactic variation does not always imply semantic divergence. To bridge this syntax-semantics gap, we introduce Code-MUE, a purely black-box framework that measures uncertainty through execution-based Semantic Interaction Graphs. Different from prior approaches that rely on superficial textual similarity, Code-MUE grounds uncertainty in observable runtime behavior, calculating the Von Neumann entropy of the solution space to quantify global semantic diversity. A large-scale empirical study across eight state-of-the-art LLMs demonstrates that Code-MUE achieves a strong negative correlation with functional correctness (Spearman's correlation up to -0.98), significantly outperforming lexical and embedding-based baselines while enabling robust risk detection and selective prediction in practical workflows.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 71 references that are in the map. First posted on arXiv 14 Jul 2026.

Builds on

47 of 71 references.

References · 47

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2017On Calibration of Modern Neural Networkscited by 4,960
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,448
  4. 2021Program Synthesis with Large Language Modelscited by 3,637
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,856
  6. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 974
  7. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 812
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 751
  9. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 210
  10. 2025Learning to Reason without External Rewardscited by 210
  11. 2021CodeNet: A Large-Scale AI for Code Dataset for Learning a Diversity of Coding Taskscited by 200
  12. 2024Kernel Language Entropy: Fine-grained Uncertainty Quantification for LLMs from Semantic Similaritiescited by 159
35 more
  1. 2023An Empirical Study of the Non-determinism of ChatGPT in Code Generationcited by 137
  2. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 121
  3. 2024LLM Hallucinations in Practical Code Generation: Phenomena, Mechanism, and Mitigationcited by 93
  4. 2023A Survey on Uncertainty Quantification Methods for Deep Learningcited by 81
  5. 2024Uncertainty Estimation and Quantification for LLMs: A Simple Supervised Approachcited by 76
  6. 2024Learning to Route LLMs with Confidence Tokenscited by 67
  7. 2024Overconfidence is Key: Verbalized Uncertainty Evaluation in Large Language and Vision-Language Modelscited by 54
  8. 2024Calibration and Correctness of Language Models for Codecited by 49
  9. 2023No Need to Lift a Finger Anymore? Assessing the Quality of Code Generation by ChatGPTcited by 47
  10. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 44
  11. 2023Look Before You Leap: An Exploratory Study of Uncertainty Measurement for Large Language Modelscited by 41
  12. 2023What Can We Learn From The Selective Prediction And Uncertainty Estimation Performance Of 523 Imagenet Classifierscited by 35
  13. 2025Uncertainty-Guided Chain-of-Thought for Code Generation with LLMscited by 32
  14. 2024InternalInspector I2: Robust Confidence Estimation in LLMs through Internal Statescited by 30
  15. 2024From Risk to Uncertainty: Generating Predictive Uncertainty Measures via Bayesian Estimationcited by 28
  16. 2025EffiBench-X: A Multi-Language Benchmark for Measuring Efficiency of LLM-Generated Codecited by 27
  17. 2020Towards Characterizing Adversarial Defects of Deep Learning Software from the Lens of Uncertaintycited by 26
  18. 2025Hallucination Detection in Large Language Models with Metamorphic Relationscited by 24
  19. 2025Thought calibration: Efficient and confident test-time scalingcited by 21
  20. 2024AcTracer: Active Testing of Large Language Model via Multi-Stage Samplingcited by 17
  21. 2025Reconsidering LLM Uncertainty Estimation Methods in the Wildcited by 17
  22. 2024Know the Unknown: An Uncertainty-Sensitive Method for LLM Instruction Tuningcited by 16
  23. 2024A Novel Characterization of the Population Area Under the Risk Coverage Curve (AURC) and Rates of Finite Sample Estimatorscited by 15
  24. 2023Do Large Language Models Pay Similar Attention Like Human Programmers When Generating Code?cited by 14
  25. 2024Drowzee: Metamorphic Testing for Fact-Conflicting Hallucination Detection in Large Language Modelscited by 14
  26. 2025Assessing Correctness in LLM-Based Code Generation via Uncertainty Estimationcited by 14
  27. 2024Uncertainty Aware Learning for Language Model Alignmentcited by 13
  28. 2024Calibration of Large Language Models on Code Summarizationcited by 12
  29. 2024Test Oracle Automation in the era of LLMscited by 11
  30. 2024Doc2OracLL: Investigating the Impact of Documentation on LLM-based Test Oracle Generationcited by 10
  31. 2025Risk Assessment Framework for Code LLMs via Leveraging Internal Statescited by 9
  32. 2025Enhancing Uncertainty Estimation in LLMs with Expectation of Aggregated Internal Beliefcited by 9
  33. 2025Towards Harmonized Uncertainty Estimation for Large Language Modelscited by 3
  34. 2025Localized Calibrated Uncertainty in Code Language Modelscited by 3
  35. 2025Multicalibration for LLM-based Code Generationcited by 2

Built on by

Not cited yet.