mlsanityData to 14 Oct 2026

Inferring Code Correctness from Specification

arXiv 2605.29822 ↗First posted 28 May 2026cs.SE cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have become integral to modern software development, enabling automated code generation at scale. However, validating the correctness of LLM-generated code remains a critical and largely unsolved challenge. Existing approaches either rely on dynamic consensus across multiple code candidates - making them costly and difficult to scale - or on static reasoning that is susceptible to dynamic bugs and order bias. In this paper, we propose TRAILS~ (Targeted Reasoning Agreement via Inputs and Specifications), an approach that grounds LLM reasoning with concrete (input, output) pairs. TRAILS~ first generates diverse test inputs via category partitioning based on the specification, then executes them against the candidate code and prompts LLMs to assess whether the resulting input-output pairs conform to the specification - without ever reasoning over the code itself. Scores are aggregated across inputs, to determines whether the program is likely correct. We evaluate TRAILS~ on two datasets, LiveCodeBench and CoCoClaNeL, across three LLMs (Qwen3Coder-30B, Devstral-Small-24B, and Olmo3.1-Instruct), comparing against HoarePrompt and a Zero-Shot Chain-of-Thought baseline. TRAILS~ improves Matthew Correlation Coefficient by up to 39% relative to Zero-Shot COT and consistently outperforms HoarePrompt. Beyond accuracy, TRAILS~ demonstrates greater stability across seeded runs, reducing sensitivity to LLM non-determinism, and assigns correct labels to a larger set of unique code samples than competing approaches.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 17 of its 31 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

17 of 31 references.

References · 17

  1. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  2. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  3. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  4. 2024Judging the Judges: A Systematic Study of Position Bias in LLM-as-a-Judgecited by 269
  5. 2023An Empirical Evaluation of Using Large Language Models for Automated Unit Test Generationcited by 194
  6. 2023ChatUniTest: A Framework for LLM-Based Test Generationcited by 96
  7. 2021TOGA: A Neural Method for Test Oracle Generationcited by 57
  8. 2024A Survey on Evaluating Large Language Models in Code Generation Taskscited by 44
  9. 2024Test-Driven Development for Code Generationcited by 34
  10. 2025Vibe Coding in Practice: Motivations, Challenges, and a Future Outlook -- a Grey Literature Reviewcited by 30
  11. 2023CAT-LM: Training Language Models on Aligned Code And Testscited by 28
  12. 2024TOGLL: Correct and Strong Test Oracle Generation with LLMscited by 24
5 more
  1. 2024Measuring the Influence of Incorrect Code on Test Generationcited by 11
  2. 2024Do LLMs generate test oracles that capture the actual or the expected program behaviour?cited by 11
  3. 2025HoarePrompt: Structural Reasoning About Program Correctness in Natural Languagecited by 7
  4. 2023A Survey on What Developers Think About Testingcited by 4
  5. 2025YATE: The Role of Test Repair in LLM-Based Unit Test Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Can LLMs Reason About Program Semantics? A Comprehensive Evaluation of LLMs on Formal Specification Inference
  2. 2025Disproving Program Equivalence with LLMs
  3. 2026CoRE: A Fine-Grained Code Reasoning Benchmark Beyond Output Prediction
  4. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code
  5. 2026An Execution-Verified Multi-Language Benchmark for Code Semantic Reasoning
  6. 2023Enhancing Large Language Models in Coding Through Multi-Perspective Self-Consistency
  7. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  8. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation