mlsanityData to 14 Oct 2026

Towards Evaluation of Implicit Software World Models in Coding LLMs

arXiv 2606.27406 ↗First posted 25 Jun 2026cs.SE cs.AIcited by 0

On the map: World models → World models for language agents

Abstract (arXiv)

Software engineering, whether performed by humans or by AI agents, requires reasoning about how software behaves. We call the internal model that supports such reasoning the software world model, and view current code-execution benchmarks as covering one well-studied slice of it -- control flow. In this paper, we take a step toward a broader evaluation by shifting the observable axis to execution resources: alongside test outcome and exception class, we predict peak memory, wall-clock time, and ranked profiler outputs at method and line granularity. We use SWE-bench Verified as the source of data to hold the test close to real-world software engineering tasks. All tested models, frontier ones included, show modest performance and brittle behaviour, suggesting a notable lack of understanding of how software is executed, as opposed to how its source code is written.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 9 of its 23 references that are in the map. First posted on arXiv 25 Jun 2026.

Builds on

9 of 23 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025CWM: An Open-Weights LLM for Research on Code Generation with World Models30 Sep

Other references · 8

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2021Program Synthesis with Large Language Modelscited by 3,637
  3. 2024Commit0: Library Generation from Scratchcited by 52
  4. 2024CodeMind: Evaluating Large Language Models for Code Reasoningcited by 32
  5. 2025CodeCrash: Exposing LLM Fragility to Misleading Natural Language in Code Reasoningcited by 14
  6. 2025BigO(Bench): Can LLMs Generate Code with Controlled Time and Space Complexity?cited by 12
  7. 2025Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settingscited by 3
  8. 2025ThrowBench: Benchmarking LLMs by Predicting Runtime Exceptionscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Benchmarks are Not Enough: RAMP for Runtime Assessing of Agentic Models in Production Systems
  2. 2025Breakpoint: Scalable evaluation of system-level reasoning in LLM code agents
  3. 2025SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models
  4. 2025Code Reasoning for Software Engineering Tasks: A Survey and A Call to Action
  5. 2025Demystifying Errors in LLM Reasoning Traces: An Empirical Study of Code Execution Simulation
  6. 2026SWE Atlas: Benchmarking Coding Agents Beyond Issue Resolution
  7. 2025LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
  8. 2026Can LLMs Reason About Runtime Behavior? A Repository-Level Dynamic Benchmark