# Environment-Aware Code Generation: How far are We?

arXiv: https://arxiv.org/abs/2601.12262 · HTML: https://mlsanity.com/paper/2601.12262/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 43 references that are in the map.
- First posted on arXiv 18 Jan 2026.

## Builds on (most cited first, 25 of 25)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Code Llama: Open Foundation Models for Code](https://mlsanity.com/paper/2308.12950/) (2023)
- [LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code](https://mlsanity.com/paper/2403.07974/) (2024)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generation](https://mlsanity.com/paper/2305.01210/) (2023)
- [Measuring Coding Challenge Competence With APPS](https://mlsanity.com/paper/2105.09938/) (2021)
- [A Survey on Large Language Models for Code Generation](https://mlsanity.com/paper/2406.00515/) (2024)
- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [BigCodeBench: Benchmarking Code Generation with Diverse Function Calls and Complex Instructions](https://mlsanity.com/paper/2406.15877/) (2024)
- [StarCoder 2 and The Stack v2: The Next Generation](https://mlsanity.com/paper/2402.19173/) (2024)
- [SWE-bench Multimodal: Do AI Systems Generalize to Visual Software Domains?](https://mlsanity.com/paper/2410.03859/) (2024)
- [Multi-lingual Evaluation of Code Generation Models](https://mlsanity.com/paper/2210.14868/) (2022)
- [CodeGemma: Open Code Models Based on Gemma](https://mlsanity.com/paper/2406.11409/) (2024)
- [A Survey on Evaluating Large Language Models in Code Generation Tasks](https://mlsanity.com/paper/2408.16498/) (2024)
- [LLMs Meet Library Evolution: Evaluating Deprecated API Usage in LLM-based Code Completion](https://mlsanity.com/paper/2406.09834/) (2024)
- [Large Language Models as Test Case Generators: Performance Evaluation and Enhancement](https://mlsanity.com/paper/2404.13340/) (2024)
- [VersiCode: Towards Version-controllable Code Generation](https://mlsanity.com/paper/2406.07411/) (2024)
- [An Empirical Study of Usages, Updates and Risks of Third-Party Libraries in Java Projects](https://mlsanity.com/paper/2002.11028/) (2020)
- [LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation](https://mlsanity.com/paper/2412.04478/) (2024)
- [ReCode: Updating Code API Knowledge with Reinforcement Learning](https://mlsanity.com/paper/2506.20495/) (2025)
- [QuCo-RAG: Quantifying Uncertainty from the Pre-training Corpus for Dynamic Retrieval-Augmented Generation](https://mlsanity.com/paper/2512.19134/) (2025)
- [GitChameleon: Unmasking the Version-Switching Capabilities of Code Generation Models](https://mlsanity.com/paper/2411.05830/) (2024)
- [OneEval: Benchmarking LLM Knowledge-intensive Reasoning over Diverse Knowledge Bases](https://mlsanity.com/paper/2506.12577/) (2025)

## Built on by (most cited first, 4 of 4)

- [CARD: Towards Conditional Design of Multi-agent Topological Structures](https://mlsanity.com/paper/2603.01089/) (2026)
- [KCoEvo: A Knowledge Graph Augmented Framework for Evolutionary Code Generation](https://mlsanity.com/paper/2603.07581/) (2026)
- [Correct Code, Vulnerable Dependencies: A Large Scale Measurement Study of LLM-Specified Library Versions](https://mlsanity.com/paper/2605.06279/) (2026)
- [Toward Executable Repository-Level Code Generation via Environment Alignment](https://mlsanity.com/paper/2604.03622/) (2026)

## Abstract (arXiv)

Recent progress in large language models (LLMs) has improved code generation, but most evaluations still test isolated, small-scale code (e.g., a single function) under default or unspecified software environments. As a result, it is unclear whether LLMs can reliably generate executable code tailored to a user's specific environment. We present the first systematic study of Environment-Aware Code Generation (EACG), where generated code must be functionally correct and directly executable under arbitrary software configurations. To enable realistic evaluation, we introduce VersiBCB, a benchmark that is multi-package, execution-verified, and deprecation-aware, capturing complex and evolving environments that prior datasets often overlook. Using VersiBCB, we investigate three complementary adaptation axes: data, parameters, and cache, and develop representative strategies for each. Our results show that current LLMs struggle with environment-specific code generation, while our adaptations improve environment compatibility and executability. These findings highlight key challenges and opportunities for deploying LLMs in practical software engineering workflows.