mlsanityData to 7 Oct 2026

AlgoVeri: An Aligned Benchmark for Verified Code Generation on Classical Algorithms

arXiv 2602.09464 ↗First posted 10 Feb 2026cs.SE cs.AI cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

Vericoding refers to the generation of formally verified code from rigorous specifications. Recent AI models show promise in vericoding, but a unified methodology for cross-paradigm evaluation is lacking. Existing benchmarks test only individual languages/tools (e.g., Dafny, Verus, and Lean) and each covers very different tasks, so the performance numbers are not directly comparable. We address this gap with AlgoVeri, a benchmark that evaluates vericoding of 77 classical algorithms in Dafny, Verus, and Lean. By enforcing identical functional contracts, AlgoVeri reveals critical capability gaps in verification systems. While frontier models achieve tractable success in Dafny (40.3% for Gemini-3 Flash), where high-level abstractions and SMT automation simplify the workflow, performance collapses under the systems-level memory constraints of Verus (24.7%) and the explicit proof construction required by Lean (7.8%). Beyond aggregate metrics, we uncover a sharp divergence in test-time compute dynamics: Gemini-3 effectively utilizes iterative repair to boost performance (e.g., tripling pass rates in Dafny), whereas GPT-OSS saturates early. Finally, our error analysis shows that language design affects the refinement trajectory: while Dafny allows models to focus on logical correctness, Verus and Lean trap models in persistent syntactic and semantic barriers. All data and evaluation code can be found at https://github.com/haoyuzhao123/algoveri.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 29 of its 45 references that are in the map. First posted on arXiv 10 Feb 2026.

Builds on

29 of 45 references.

References · 29

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2023Code Llama: Open Foundation Models for Codecited by 2,105
  3. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  4. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  5. 2023Llemma: An Open Language Model For Mathematicscited by 385
  6. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Modelscited by 282
  7. 2025DeepSeek-Prover-V2: Advancing Formal Mathematical Reasoning via Reinforcement Learning for Subgoal Decompositioncited by 270
  8. 2024DeepSeek-Prover-V1.5: Harnessing Proof Assistant Feedback for Reinforcement Learning and Monte-Carlo Tree Searchcited by 189
  9. 2025Kimina-Prover Preview: Towards Large Formal Reasoning Models with Reinforcement Learningcited by 173
  10. 2025Goedel-Prover-V2: Scaling Formal Theorem Proving with Scaffolded Data Synthesis and Self-Correctioncited by 140
  11. 2025Seed-Prover: Deep and Broad Reasoning for Automated Theorem Provingcited by 96
  12. 2025BFS-Prover: Scalable Best-First Tree Search for LLM-based Automatic Theorem Provingcited by 82
17 more
  1. 2023An In-Context Learning Agent for Formal Theorem-Provingcited by 78
  2. 2025STP: Self-play LLM Theorem Provers with Iterative Conjecturing and Provingcited by 77
  3. 2024Towards AI-Assisted Synthesis of Verified Dafny Methodscited by 58
  4. 2024DafnyBench: A Benchmark for Formal Software Verificationcited by 54
  5. 2023Verus: Verifying Rust Programs using Linear Ghost Types (extended version)cited by 50
  6. 2024InternLM2.5-StepProver: Advancing Automated Theorem Proving via Critic-Guided Searchcited by 34
  7. 2024HunyuanProver: A Scalable Data Synthesis Framework and Guided Tree Search for Automated Theorem Provingcited by 32
  8. 2024Automated Proof Generation for Rust Code via Self-Evolutioncited by 31
  9. 2025VERINA: Benchmarking Verifiable Code Generationcited by 30
  10. 2025CLEVER: A Curated Benchmark for Formally Verified Code Generationcited by 27
  11. 2025Security and Quality in LLM-Generated Code: A Multi-Language, Multi-Model Analysiscited by 18
  12. 2025A benchmark for vericoding: formally verified program synthesiscited by 16
  13. 2025VeruSAGE: A Study of Agent-Based Verification for Rust Systemscited by 11
  14. 2025VerifyThisBench: Generating Code, Specifications, and Proofs All at Oncecited by 9
  15. 2025Re:Form -- Reducing Human Annotations in Scalable Formal Software Verification with RL in LLMs: A Preliminary Study on Dafnycited by 9
  16. 2020Detecting Critical Bugs in SMT Solvers Using Blackbox Mutational Fuzzingcited by 4
  17. 2025Can LLMs Enable Verification in Mainstream Programming?cited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 11.

11 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026Automating Formal Verification with Agent-Guided Tree Search
  2. 2025VeriEquivBench: An Equivalence Score for Ground-Truth-Free Evaluation of Formally Verifiable Code
  3. 2026AxDafny: Agentic Verified Code Generation in Dafny
  4. 2024AlphaVerus: Bootstrapping Formally Verified Code Generation through Self-Improving Translation and Treefinement
  5. 2026WybeCoder: Verified Imperative Code Generation
  6. 2024Evaluating the Ability of Large Language Models to Generate Verifiable Specifications in VeriFast
  7. 2026SWE-Proof: Can Language Models Resolve Real-World Issues with Machine-Checked Proofs?
  8. 2025RAG-Verus: Repository-Level Program Verification with LLMs using Retrieval Augmented Generation