mlsanityData to 7 Oct 2026

DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents

arXiv 2511.20709 ↗First posted 24 Nov 2025cs.SE cs.AI cs.CRcited by 1

On the map: LLM agents and tool use → Agent threats, guardrails and governance

Abstract (arXiv)

Large language models (LLMs) and LLM-based coding agents are now used to generate code from natural-language specifications, yet ensuring such code is both functionally correct and secure remains a challenge. We present DualGauge, the first fully automated framework for jointly evaluating correctness and security of specification-only code generation, supported by DualGauge-Bench, a language-agnostic benchmark of 307 coding tasks each paired with functional and security tests derived from the same specification. Evaluating 10 representative LLMs across Python, C++, and JavaScript, we find that functional correctness substantially overestimates reliable code generation: even the strongest model remains below 15% joint security-functionality success in every language. Common model-side factors--scale, extended thinking, quantization, instruction tuning, and code specialization--do not reliably improve joint performance, suggesting secure-and-correct code generation does not simply emerge from stronger coding capability. Evaluation of 3 leading agentic coding systems (Codex, OpenHands, and Claude Code) shows that iterative scaffolding provides no advantage over direct (LLM-based) generation on specification-only tasks. A qualitative audit reveals failures concentrate at the output contract boundary and in guards that exist but are insufficient--patterns that only joint benchmarking reliably exposes.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 26 of its 47 references that are in the map. First posted on arXiv 24 Nov 2025.

Builds on

26 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025SecRepoBench: Benchmarking Code Agents for Secure Code Completion in Real-World Repositories29 Apr

Other references · 25

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  2. 2021Program Synthesis with Large Language Modelscited by 3,509
  3. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  4. 2021Measuring Coding Challenge Competence With APPScited by 879
  5. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  6. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 242
  7. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  8. 2023Large Language Models for Code: Security Hardening and Adversarial Testingcited by 149
  9. 2023Purple Llama CyberSecEval: A Secure Coding Benchmark for Language Modelscited by 130
  10. 2022Lost at C: A User Study on the Security Implications of Large Language Model Code Assistantscited by 87
  11. 2024Instruction Tuning for Secure Code Generationcited by 63
  12. 2025CWEval: Outcome-driven Evaluation on Functionality and Security of LLM Code Generationcited by 55
13 more
  1. 2023CodeLMSec Benchmark: Systematically Evaluating and Finding Security Vulnerabilities in Black-Box Code Language Modelscited by 52
  2. 2024You Name It, I Run It: An LLM Agent to Execute Tests of Arbitrary Projectscited by 45
  3. 2025BaxBench: Can LLMs Generate Correct and Secure Backends?cited by 38
  4. 2023Security Weaknesses of Copilot-Generated Code in GitHub Projects: An Empirical Studycited by 36
  5. 2024Constrained Decoding for Secure Code Generationcited by 35
  6. 2023Assessing the Security of GitHub Copilot Generated Code -- A Targeted Replication Studycited by 24
  7. 2024ProSec: Fortifying Code LLMs with Proactive Security Alignmentcited by 24
  8. 2025SetupBench: Assessing Software Engineering Agents' Ability to Bootstrap Development Environmentscited by 18
  9. 2025SafeGenBench: A Benchmark Framework for Security Vulnerability Detection in LLM-Generated Codecited by 17
  10. 2025PurpCode: Reasoning for Safer Code Generationcited by 17
  11. 2023A User-centered Security Evaluation of Copilotcited by 14
  12. 2023Enhancing Large Language Models for Secure Code Generation: A Dataset-driven Study on Vulnerability Mitigationcited by 14
  13. 2025CXXCrafter: An LLM-Based Agent for Automated C/C++ Open Source Software Buildingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Local LLM Agents as Vulnerable Runtimes:A Source-Code Audit of the Agent Runtime Layercited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025SCGAgent: Recreating the Benefits of Reasoning Models for Secure Code Generation with Agentic Workflows
  2. 2026SoK: AI Secure Code Generation: Progress, Pitfalls, and Paths Forward
  3. 2024AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing
  4. 2025TypePilot: Leveraging the Scala Type System for Secure LLM-generated Code
  5. 2026Think Before You Code: Dual Reasoning for the NLSafety-Utility Trade-Off in LLM Code Generation
  6. 2025Beyond Synthetic Benchmarks: Evaluating LLM Performance on Real-World Class-Level Code Generation
  7. 2025Rethinking the Evaluation of Secure Code Generation
  8. 2026Improving LLM-Assisted Secure Code Generation through Retrieval-Augmented-Generation and Multi-Tool Feedback