mlsanityData to 14 Oct 2026

The Self-Execution Benchmark: Measuring LLMs' Attempts to Overcome Their Lack of Self-Execution

arXiv 2508.12277 ↗First posted 17 Aug 2025cs.CL cs.AIcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Large language models (LLMs) are commonly evaluated on tasks that test their knowledge or reasoning abilities. In this paper, we explore a different type of evaluation: whether an LLM can predict aspects of its own responses. Since LLMs lack the ability to execute themselves, we introduce the Self-Execution Benchmark, which measures a model's ability to anticipate properties of its output, such as whether a question will be difficult for it, whether it will refuse to answer, or what kinds of associations it is likely to produce. Our experiments show that models generally perform poorly on this benchmark, and that increased model size or capability does not consistently lead to better performance. These results suggest a fundamental limitation in how LLMs represent and reason about their own behavior.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 9 of its 18 references that are in the map. First posted on arXiv 17 Aug 2025.

Builds on

9 of 18 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025Reliable and Efficient Amortized Model-based Evaluation17 Mar

Other references · 8

  1. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  2. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  3. 2023Let's Verify Step by Stepcited by 3,789
  4. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  5. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  6. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  7. 2024Me, Myself, and AI: The Situational Awareness Dataset (SAD) for LLMscited by 84
  8. 2024Code Comprehension then Auditing for Unsupervised LLM Evaluationcited by 32

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025The Illusion of Diminishing Returns: Measuring Long Horizon Execution in LLMs
  2. 2024Enhancing LLM Evaluations: The Garbling Trick
  3. 2024Internal Consistency and Self-Feedback in Large Language Models: A Survey
  4. 2024Mind the Gap: Examining the Self-Improvement Capabilities of Large Language Models
  5. 2025Self-Evolving Critique Abilities in Large Language Models
  6. 2024Beyond Accuracy: Evaluating the Reasoning Behavior of Large Language Models -- A Survey
  7. 2024Self-Contrast: Better Reflection Through Inconsistent Solving Perspectives
  8. 2024metabench -- A Sparse Benchmark of Reasoning and Knowledge in Large Language Models