mlsanityData to 7 Oct 2026

Multicalibration for LLM-based Code Generation

arXiv 2512.08810 ↗First posted 9 Dec 2025cs.SE cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

As AI-based code generation becomes widespread, researchers are investigating the calibration of code LLMs - ensuring their confidence scores faithfully represent the true likelihood of code correctness. To do so, we investigate multicalibration, which can capture additional factors about a coding problem, such as complexity, code length, or programming language used. We study four multicalibration approaches on three function synthesis benchmarks, using latest-generation code LLMs (Qwen3 Coder, GPT-OSS, DeepSeek-R1-Distill). Our results demonstrate that multicalibration can yield distinct improvements over both uncalibrated token likelihoods (+1.03 in skill score) and baseline calibrations (+0.37 in skill score). We study the influence of the aforementioned factors in ablations, and make our dataset (consisting of code generations, likelihoods, and correctness labels) available for future research on code LLM calibration.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 31 of its 56 references that are in the map. First posted on arXiv 9 Dec 2025.

Builds on

31 of 56 references.

References · 31

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2021Program Synthesis with Large Language Modelscited by 3,509
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 1,906
  7. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  8. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  9. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  10. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  11. 2023Large Language Models for Software Engineering: A Systematic Literature Reviewcited by 389
  12. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 253
19 more
  1. 2018Multiaccuracy: Black-Box Post-Processing for Fairness in Classificationcited by 221
  2. 2020Unsupervised Quality Estimation for Neural Machine Translationcited by 210
  3. 2025Deep Think with Confidencecited by 199
  4. 2023Shifting Attention to Relevance: Towards the Predictive Uncertainty Quantification of Free-Form Large Language Modelscited by 147
  5. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  6. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  7. 2023CodeBERTScore: Evaluating Code Generation with Pretrained Models of Codecited by 101
  8. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  9. 2022Loss Minimization through the Lens of Outcome Indistinguishabilitycited by 61
  10. 2024Bugs in Large Language Models Generated Code: An Empirical Studycited by 59
  11. 2023On Early Detection of Hallucinations in Factual Question Answeringcited by 54
  12. 2024Calibration and Correctness of Language Models for Codecited by 49
  13. 2023Large Language Models and Simple, Stupid Bugscited by 45
  14. 2024CodeJudge: Evaluating Code Generation with Large Language Modelscited by 41
  15. 2024A Deep Dive into Large Language Models for Automated Bug Localization and Repaircited by 39
  16. 2025MCTS-Judge: Test-Time Scaling in LLM-as-a-Judge for Code Correctness Evaluationcited by 32
  17. 2025Correctness Assessment of Code Generated by Large Language Models Using Internal Representationscited by 17
  18. 2024When is Multicalibration Post-Processing Necessary?cited by 16
  19. 2025From Code to Courtroom: LLMs as the New Software Judgescited by 16

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Beyond Semantic Relevance: Counterfactual Risk Minimization for Robust Retrieval-Augmented Generationcited by 1
  2. 2026Code-MUE: Measuring Code LLMs' Uncertainty through Execution-based Semantic Interaction Graphscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Preserving LLM Capabilities through Calibration Data Curation: From Analysis to Optimization
  2. 2025Localized Calibrated Uncertainty in Code Language Models
  3. 2024Does Alignment Tuning Really Break LLMs' Internal Confidence?
  4. 2026Enhancing LLM-Based Code Translation with Verified Multi-Semantic Representations ↗
  5. 2025LLM4VV: Evaluating Cutting-Edge LLMs for Generation and Evaluation of Directive-Based Parallel Programming Model Compiler Tests ↗
  6. 2025Holistic Evaluation of State-of-the-Art LLMs for Code Generation
  7. 2024Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks
  8. 2024Calibrating Long-form Generations from Large Language Models