mlsanityData to 14 Oct 2026

Code-Guided Reasoning for Small Language Models: Evaluating Executable MCQA Scaffolds

arXiv 2605.18827 ↗First posted 12 May 2026cs.IR cs.LG cs.PLcited by 0

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Multiple-choice QA benchmarks usually evaluate small language models (SLMs) as direct answerers, but deployed language-model systems increasingly rely on external scaffolds such as tools, code, and repeated model calls. We introduce Code-Guided Reasoning (CGR), an evaluation protocol and generated-program resource for measuring when executable reasoning scaffolds improve SLM performance on MCQA tasks. CGR standardizes six components: a normalized item interface, a direct solver prompt, a generator prompt, a Python scaffold, solver-call and extraction helpers, and a three-channel result record. On 20,498 retained result rows from a locally prepared MCQA bundle and six metadata-registered solver models, the observed non-zero-baseline partition shows 66.21% macro assisted accuracy versus 38.11% direct accuracy, a +28.10 percentage-point difference with a pair-bootstrap interval of [20.32, 36.43]. Under a stricter Ab > 30% direct-signal gate, the macro difference is +14.11 points. These estimates are descriptive. Assisted inference uses a larger solver-call budget, answer extraction is brittle, Time-MQA contains the observed regressions, and some generated programs violate the no-hard-coding instruction. CGR provides the trace package needed to interpret these results, including direct, assisted, and generator-side answers, partition definitions, generated programs, response metadata, and audits.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 6 of its 26 references that are in the map. First posted on arXiv 12 May 2026.

Builds on

6 of 26 references.

References · 6

  1. 2021Measuring Coding Challenge Competence With APPScited by 902
  2. 2025Humanity's Last Examcited by 491
  3. 2025SuperGPQA: Scaling LLM Evaluation across 285 Graduate Disciplinescited by 79
  4. 2025ReasonFlux: Hierarchical LLM Reasoning via Scaling Thought Templatescited by 63
  5. 2024Improving Physics Reasoning in Large Language Models Using Mixture of Refinement Agentscited by 24
  6. 2024LLMs for Doctors: Leveraging Medical LLMs to Assist Doctors, Not Replace Themcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
  2. 2025Enhancing Reasoning Capabilities of Small Language Models with Blueprints and Prompt Template Search
  3. 2025Guiding Reasoning in Small Language Models with LLM Assistance
  4. 2026CodeEval: A pedagogical approach for targeted evaluation of code-trained Large Language Models
  5. 2025Slm-mux: Orchestrating small language models for reasoning
  6. 2025SLR: Automated Synthesis for Scalable Logical Reasoning
  7. 2025ReTraceQA: Evaluating Reasoning Traces of Small Language Models in Commonsense Question Answering
  8. 2025rStar-Coder: Scaling Competitive Code Reasoning with a Large-Scale Verified Dataset