mlsanityData to 7 Oct 2026

Targeted Tests for LLM Reasoning: An Audit-Constrained Protocol

arXiv 2605.11599 ↗First posted 12 May 2026cs.LGcited by 0

On the map: LLM reasoning → Prompting and chain-of-thought reasoning

Abstract (arXiv)

Fixed reasoning benchmarks evaluate canonical prompts, but semantically valid changes in presentation can still change model behavior. Studies of prompt variation can reveal such failures, but without audit they can mix genuine model errors with invalid perturbations, extraction artifacts, and unmatched search procedures. We propose an audit-constrained protocol for targeted reasoning evaluation. Prompt variants are generated from a finite component grammar, rendered deterministically, evaluated under a fixed query budget, and counted as model errors only after semantic and extraction audit. Within this protocol we instantiate Component-Adaptive Prompt Sampling (CAPS), a score-based sampler over prompt components, and compare it with equal-budget uniform component sampling under the same task bank, renderer, model interface, decoding settings, and audit procedure. Across three audited slices, the protocol identifies confirmed model-error prompt keys while excluding formatting and extraction artifacts, but matched comparisons do not show that CAPS improves audited yield or unique prompt-key discovery over uniform sampling. The contribution is methodological: targeted prompt variation can be studied under a reconstructable, reviewable, budget-matched protocol, and proxy-guided policies should be judged by audited yield rather than raw mismatch counts or selected examples alone.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Prompting and chain-of-thought reasoning, one of 17 directions in LLM reasoning. Builds on 8 of its 12 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

8 of 12 references.

Closest ancestors · 2

Same direction, newest first

  1. 2022Large Language Models are Zero-Shot Reasoners24 May
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Models28 Jan

Other references · 6

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  4. 2022Red Teaming Language Models with Language Modelscited by 983
  5. 2020Beyond Accuracy: Behavioral Testing of NLP models with CheckListcited by 979
  6. 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 709

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025You Don't Need Prompt Engineering Anymore: The Prompting Inversion
  2. 2026Reflective Prompt Tuning through Language Model Function-Calling
  3. 2025DETAIL Matters: Measuring the Impact of Prompt Specificity on Reasoning in Large Language Models
  4. 2025Prompting Test-Time Scaling Is A Strong LLM Reasoning Data Augmentation
  5. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMs
  6. 2025Self-Supervised Prompt Optimization
  7. 2024A Looming Replication Crisis in Evaluating Behavior in Language Models? Evidence and Solutions
  8. 2026Adaptive Prompt Structure Factorization: A Framework for Self-Discovering and Optimizing Compositional Prompt Programs