# From Stochastic Answers to Verifiable Reasoning: Interpretable Decision-Making with LLM-Generated Code

arXiv: https://arxiv.org/abs/2603.13287 · HTML: https://mlsanity.com/paper/2603.13287/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning.
- Builds on 4 of its 11 references that are in the map.
- First posted on arXiv 28 Feb 2026.

## Placed in

- LLM reasoning / [Reasoning benchmarks and evaluation](https://mlsanity.com/lr/reasoning-benchmarks-and-evaluation/)

## Builds on (most cited first, 4 of 4)

- [LLM-based feature generation from text for interpretable machine learning](https://mlsanity.com/paper/2409.07132/) (2024)
- [GPTree: Towards Explainable Decision-Making via LLM-powered Decision Trees](https://mlsanity.com/paper/2411.08257/) (2024)
- [VCBench: Benchmarking LLMs in Venture Capital](https://mlsanity.com/paper/2509.14448/) (2025)
- [Sparse Focus Network for Multi-Source Remote Sensing Data Classification](https://mlsanity.com/paper/2406.01245/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are increasingly used for high-stakes decision-making, yet existing approaches struggle to reconcile scalability, interpretability, and reproducibility. Black-box models obscure their reasoning, while recent LLM-based rule systems rely on per-sample evaluation, causing costs to scale with dataset size and introducing stochastic, hallucination-prone outputs. We propose reframing LLMs as code generators rather than per-instance evaluators. A single LLM call generates executable, human-readable decision logic that runs deterministically over structured data, eliminating per-sample LLM queries while enabling reproducible and auditable predictions. We combine code generation with automated statistical validation using precision lift, binomial significance testing, and coverage filtering, and apply cluster-based gap analysis to iteratively refine decision logic without human annotation. We instantiate this framework in venture capital founder screening, a rare-event prediction task with strong interpretability requirements. On VCBench, a benchmark of 4,500 founders with a 9% base success rate, our approach achieves 37.5% precision and an F0.5 score of 25.0%, outperforming GPT-4o (at 30.0% precision and an F0.5 score of 25.7%) while maintaining full interpretability. Each prediction traces to executable rules over human-readable attributes, demonstrating verifiable and interpretable LLM-based decision-making in practice.