mlsanityData to 7 Oct 2026

Correctness isnt Efficiency: Runtime Memory Divergence in LLM-Generated Code

arXiv 2601.01215 ↗First posted 3 Jan 2026cs.SE cs.AIcited by 4

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) can generate programs that pass unit tests, but passing tests does not guarantee reliable runtime behavior. We find that different correct solutions to the same task can show very different memory and performance patterns, which can lead to hidden operational risks. We present a framework to measure execution-time memory stability across multiple correct generations. At the solution level, we introduce Dynamic Mean Pairwise Distance (DMPD), which uses Dynamic Time Warping to compare the shapes of memory-usage traces after converting them into Monotonic Peak Profiles (MPPs) to reduce transient noise. Aggregating DMPD across tasks yields a model-level Model Instability Score (MIS). Experiments on BigOBench and CodeContests show substantial runtime divergence among correct solutions. Instability often increases with higher sampling temperature even when pass@1 improves. We also observe correlations between our stability measures and software engineering indicators such as cognitive and cyclomatic complexity, suggesting links between operational behavior and maintainability. Our results support stability-aware selection among passing candidates in CI/CD to reduce operational risk without sacrificing correctness. Artifacts are available.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 13 of its 49 references that are in the map. First posted on arXiv 3 Jan 2026.

Builds on

13 of 49 references.

References · 13

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2021Program Synthesis with Large Language Modelscited by 3,509
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,427
  6. 2024BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructionscited by 527
  7. 2020CodeBLEU: a Method for Automatic Evaluation of Code Synthesiscited by 468
  8. 2021Asleep at the Keyboard? Assessing the Security of GitHub Copilot's Code Contributionscited by 417
  9. 2022Do Users Write More Insecure Code with AI Assistants?cited by 186
  10. 2023Is ChatGPT the Ultimate Programming Assistant -- How far is it?cited by 72
  11. 2024Mercury: A Code Efficiency Benchmark for Code Large Language Modelscited by 34
  12. 2024Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distancecited by 26
1 more
  1. 2025GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.