# Dynamic Stability of LLM-Generated Code

arXiv: https://arxiv.org/abs/2511.07463 · HTML: https://mlsanity.com/paper/2511.07463/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 34 references that are in the map.
- First posted on arXiv 7 Nov 2025.

## Builds on (most cited first, 13 of 13)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [CodeBLEU: a Method for Automatic Evaluation of Code Synthesis](https://mlsanity.com/paper/2009.10297/) (2020)
- [A Systematic Evaluation of Large Language Models of Code](https://mlsanity.com/paper/2202.13169/) (2022)
- [Neurosymbolic AI: The 3rd Wave](https://mlsanity.com/paper/2012.05876/) (2020)
- [The Next Decade in AI: Four Steps Towards Robust Artificial Intelligence](https://mlsanity.com/paper/2002.06177/) (2020)
- [Is ChatGPT the Ultimate Programming Assistant -- How far is it?](https://mlsanity.com/paper/2304.11938/) (2023)
- [A Survey on Large Language Models for Software Engineering](https://mlsanity.com/paper/2312.15223/) (2023)
- [Revisiting Code Similarity Evaluation with Abstract Syntax Tree Edit Distance](https://mlsanity.com/paper/2404.08817/) (2024)
- [GREEN-CODE: Learning to Optimize Energy Efficiency in LLM-based Code Generation](https://mlsanity.com/paper/2501.11006/) (2025)
- [Does Acceleration Cause Hidden Instability in Vision Language Models? Uncovering Instance-Level Divergence Through a Large-Scale Empirical Study](https://mlsanity.com/paper/2503.06794/) (2025)

## Built on by (most cited first, 4 of 4)

- [Can Large Language Models Implement Agent-Based Models? An ODD-based Replication Study](https://mlsanity.com/paper/2602.10140/) (2026)
- [SQLStructEval: Structural Evaluation of LLM Text-to-SQL Generation](https://mlsanity.com/paper/2604.06736/) (2026)
- [Algorithmic algorithm development with LLMs: A Case Study on LLM-Usage for Contraction Order Optimization in Tensor Networks](https://mlsanity.com/paper/2606.01975/) (2026)
- [LLM Post-Training as Brownfield Maintenance: An Industrial Perspective on Dataware Engineering](https://mlsanity.com/paper/2608.31102/) (2026)

## Abstract (arXiv)

Current evaluations of LLMs for code generation emphasize functional correctness, overlooking the fact that functionally correct solutions can differ significantly in algorithmic complexity. For instance, an (O(n2)) versus (O(n log n)) sorting algorithm may yield similar output but incur vastly different performance costs in production. This discrepancy reveals a critical limitation in current evaluation methods: they fail to capture the behavioral and performance diversity among correct solutions. To address this, we introduce a principled framework for evaluating the dynamic stability of generated code. We propose two metrics derived from opcode distributions: Static Canonical Trace Divergence (SCTD), which captures algorithmic structure diversity across generated solutions, and Dynamic Canonical Trace Divergence (DCTD), which quantifies runtime behavioral variance. Their ratio, the Behavioral Expression Factor (BEF), serves as a diagnostic signal: it indicates critical runtime instability when BEF ll 1 and functional redundancy when BEF gg 1. Empirical results on BigOBench and CodeContests show that state-of-the-art LLMs exhibit significant algorithmic variance even among functionally correct outputs. Notably, increasing sampling temperature improves pass@1 rates but degrades stability, revealing an unrecognized trade-off: searching for correct solutions in diverse output spaces introduces a "penalty of instability" between correctness and behavioral consistency. Our findings call for stability-aware objectives in code generation and new benchmarks with asymptotic test cases for robust, real-world LLM evaluation.