# Coding in a Bubble? Evaluating LLMs in Resolving Context Adaptation Bugs During Code Adaptation

arXiv: https://arxiv.org/abs/2601.06497 · HTML: https://mlsanity.com/paper/2601.06497/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 74 references that are in the map.
- First posted on arXiv 10 Jan 2026.

## Builds on (most cited first, 30 of 32)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Grounded Copilot: How Programmers Interact with Code-Generating Models](https://mlsanity.com/paper/2206.15000/) (2022)
- [Automating Code Review Activities by Large-Scale Pre-training](https://mlsanity.com/paper/2203.09095/) (2022)
- [DebugBench: Evaluating Debugging Capability of Large Language Models](https://mlsanity.com/paper/2401.04621/) (2024)
- [Calibration and Correctness of Language Models for Code](https://mlsanity.com/paper/2402.02047/) (2024)
- [Reasoning Runtime Behavior of a Program with LLM: How Far Are We?](https://mlsanity.com/paper/2403.16437/) (2024)
- [Neural Program Repair by Jointly Learning to Localize and Repair](https://mlsanity.com/paper/1904.01720/) (2019)
- [A Survey on Deep Learning for Software Engineering](https://mlsanity.com/paper/2011.14597/) (2020)
- [Search-Based LLMs for Code Optimization](https://mlsanity.com/paper/2408.12159/) (2024)
- [CCT5: A Code-Change-Oriented Pre-Trained Model](https://mlsanity.com/paper/2305.10785/) (2023)
- [Stack Overflow Considered Harmful? The Impact of Copy&Paste on Android Application Security](https://mlsanity.com/paper/1710.03135/) (2017)
- [Aligning the Objective of LLM-based Program Repair](https://mlsanity.com/paper/2404.08877/) (2024)
- [Toxic Code Snippets on Stack Overflow](https://mlsanity.com/paper/1806.07659/) (2018)
- [SmartPaste: Learning to Adapt Source Code](https://mlsanity.com/paper/1705.07867/) (2017)
- [Analyzing and Supporting Adaptation of Online Code Examples](https://mlsanity.com/paper/1905.12111/) (2019)
- [Unseen Horizons: Unveiling the Real Capability of LLM Code Generation Beyond the Familiar](https://mlsanity.com/paper/2412.08109/) (2024)
- [From Query to Usable Code: An Analysis of Stack Overflow Code Snippets](https://mlsanity.com/paper/1605.04464/) (2016)
- [APIzation: Generating Reusable APIs from StackOverflow Code Snippets](https://mlsanity.com/paper/2109.01138/) (2021)
- [Stack Overflow in Github: Any Snippets There?](https://mlsanity.com/paper/1705.01198/) (2017)
- [Optimising the Fit of Stack Overflow Code Snippets into Existing Code](https://mlsanity.com/paper/2004.07663/) (2020)
- [Towards Exploring the Code Reuse from Stack Overflow during Software Development](https://mlsanity.com/paper/2204.12711/) (2022)
- [AI Tool Use and Adoption in Software Development by Individuals and Organizations: A Grounded Theory Study](https://mlsanity.com/paper/2406.17325/) (2024)
- [Stack Overflow: A Code Laundering Platform?](https://mlsanity.com/paper/1703.03897/) (2017)
- [AdaptivePaste: Code Adaptation through Learning Semantics-aware Variable Usage Representations](https://mlsanity.com/paper/2205.11023/) (2022)
- [On the Possibility of Breaking Copyleft Licenses When Reusing Code Generated by ChatGPT](https://mlsanity.com/paper/2502.05023/) (2025)
- [NCQ: Code reuse support for node.js developers](https://mlsanity.com/paper/2101.00756/) (2021)
- [On Code Reuse from StackOverflow: An Exploratory Study on Jupyter Notebook](https://mlsanity.com/paper/2302.11732/) (2023)
- [Are Human Rules Necessary? Generating Reusable APIs with CoT Reasoning and In-Context Learning](https://mlsanity.com/paper/2405.03509/) (2024)

## Abstract (arXiv)

Code adaptation is a fundamental but challenging task in software development, requiring developers to modify existing code for new contexts. A key challenge is to resolve Context Adaptation Bugs (CtxBugs), which occurs when code correct in its original context violates constraints in the target environment. Unlike isolated bugs, CtxBugs cannot be resolved through local fixes and require cross-context reasoning to identify semantic mismatches. Overlooking them may lead to critical failures in adaptation. Although Large Language Models (LLMs) show great potential in automating code-related tasks, their ability to resolve CtxBugs remains a significant and unexplored obstacle to their practical use in code adaptation. To bridge this gap, we propose CtxBugGen, a novel framework for generating CtxBugs to evaluate LLMs. Its core idea is to leverage LLMs' tendency to generate plausible but context-free code when contextual constraints are absent. The framework generates CtxBugs through a four-step process to ensure their relevance and validity: (1) Adaptation Task Selection, (2) Task-specific Perturbation,(3) LLM-based Variant Generation and (4) CtxBugs Identification. Based on the benchmark constructed by CtxBugGen, we conduct an empirical study with four state-of-the-art LLMs. Our results reveal their unsatisfactory performance in CtxBug resolution. The best performing LLM, Kimi-K2, achieves 55.93% on Pass@1 and resolves just 52.47% of CtxBugs. The presence of CtxBugs degrades LLMs' adaptation performance by up to 30%. Failure analysis indicates that LLMs often overlook CtxBugs and replicate them in their outputs. Our study highlights a critical weakness in LLMs' cross-context reasoning and emphasize the need for new methods to enhance their context awareness for reliable code adaptation.