mlsanityData to 14 Oct 2026

CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoning

arXiv 2504.14119 ↗First posted 19 Apr 2025cs.AI cs.SEcited by 14

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Large Language Models (LLMs) have recently demonstrated strong capabilities in code-related tasks, but their robustness in code reasoning under perturbations remains underexplored. We introduce CodeCrash, a stress-testing framework with 1,279 questions from CruxEval and LiveCodeBench, designed to evaluate reasoning reliability under structural perturbations and misleading natural language (NL) contexts. Through a systematic evaluation of 17 LLMs, we find that models often shortcut reasoning by over-relying on NL cues, leading to an average performance degradation of 23.2% in output prediction tasks. Even with Chain-of-Thought reasoning, models on average still have a 13.8% drop due to distractibility and rationalization, revealing a lack of critical reasoning capability to distinguish the actual code behaviors. While Large Reasoning Models with internal reasoning mechanisms improve robustness by fostering critical thinking, plausible yet incorrect hints can trigger pathological self-reflection, causing 2-3 times token consumption and even catastrophic cognitive dissonance in extreme cases for QwQ-32B. We refer to this phenomenon as Reasoning Collapse. CodeCrash provides a rigorous benchmark for evaluating robustness in code reasoning, guiding future research and development toward more reliable and resilient models.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 31 of its 65 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

31 of 65 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025OverThink: Slowdown Attacks on Reasoning LLMs4 Feb
  2. 2024Qwen2.5-Coder Technical Report18 Sep
  3. 2024NExT: Teaching Large Language Models to Reason about Code Execution23 Apr
  4. 2024Reasoning Runtime Behavior of a Program with LLM: How Far Are We?25 Mar

Other references · 27

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2021Measuring Coding Challenge Competence With APPScited by 902
  9. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Modelscited by 886
  10. 2024DeepSeek-V3 Technical Reportcited by 811
  11. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  12. 2025The Danger of Overthinking: Examining the Reasoning-Action Dilemma in Agentic Taskscited by 152
15 more
  1. 2022Large Language Models Meet NL2Code: A Surveycited by 138
  2. 2022ReCode: Robustness Evaluation of Code Generation Modelscited by 103
  3. 2023CodeScore: Evaluating Code Generation by Learning Code Executioncited by 67
  4. 2023On the Robustness of Code Generation Techniques: An Empirical Study on GitHub Copilotcited by 64
  5. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 51
  6. 2023On Robustness of Prompt-based Semantic Parsing with Large Pre-trained Language Model: An Empirical Study on Codexcited by 41
  7. 2022StructCoder: Structure-Aware Transformer for Code Generationcited by 34
  8. 2023Code Execution with Pre-trained Language Modelscited by 34
  9. 2023Reinforcement Learning from Automatic Feedback for High-Quality Unit Test Generationcited by 21
  10. 2024NLPerturbator: Studying the Robustness of Code LLMs to Natural Language Variationscited by 20
  11. 2024Transforming Software Development: Evaluating the Efficiency and Challenges of GitHub Copilot in Real-World Projectscited by 18
  12. 2024Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?cited by 18
  13. 2024MdEval: Massively Multilingual Code Debuggingcited by 16
  14. 2024Large Language Models as Code Executors: An Exploratory Studycited by 12
  15. 2024CSA-Trans: Code Structure Aware Transformer for ASTcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 7.

14 papers.

Sort:

By area:

  1. 2025DesignBench: A Comprehensive Benchmark for MLLM-based Front-end Code Generationcited by 45
  2. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  3. 2025CoRe: Benchmarking LLMs Code Reasoning Capabilities through Static Analysis Taskscited by 24
  4. 2026SWE-Chain: Benchmarking Coding Agents on Chained Release-Level Package Upgradescited by 7
  5. 2025IRG-MotionLLM: Interleaving Motion Generation, Assessment and Refinement for Text-to-Motion Generationcited by 6
  6. 2025Conflict-Aware Fusion: Mitigating Logic Inertia in Large Language Models via Structured Cognitive Priorscited by 5
  7. 2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 5
  8. 2025Refining Critical Thinking in LLM Code Generation: A Faulty Premise-based Evaluation Frameworkcited by 1
  9. 2025TREAT: A Code LLMs Trustworthiness / Reliability Evaluation and Testing Frameworkcited by 1
  10. 2026Themis: Training Robust Multilingual Code Reward Models for Flexible Multi-Criteria Scoringcited by 1
  11. 2026The Invisible Lottery: How Subtle Cues Steer Algorithm Choice in LLM Code Generationcited by 1
  12. 2026Towards Evaluation of Implicit Software World Models in Coding LLMscited by 0
2 more
  1. 2026Wrong Design Intent Is Worse Than Never Conditioning: A Derangement-Control Diagnosis of Header Conditioning in CAD Program Completioncited by 0
  2. 2026ALIBI: Adaptive Agentic Attacks on LLM-Based Vulnerability Detectors via Adversarial Code Commentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Unveiling the Magic of Code Reasoning through Hypothesis Decomposition and Amendment
  2. 2025On Code-Induced Reasoning in LLMs
  3. 2025The Valley of Code Reasoning: Scaling Knowledge Distillation of Large Language Models
  4. 2026Fragile Thoughts: How Large Language Models Handle Chain-of-Thought Perturbations
  5. 2026Reasoning Shift: How Context Silently Shortens LLM Reasoning
  6. 2026Exploring Extrinsic and Intrinsic Properties for Effective Reasoning with Code Interpreter
  7. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
  8. 2025Are Large Language Models Robust in Understanding Code Against Semantics-Preserving Mutations?