# From Checker to Forecaster: Code-Owned Evaluation of Model-Generated Strategic Routes Under Delayed Ground Truth

arXiv: https://arxiv.org/abs/2607.10972 · HTML: https://mlsanity.com/paper/2607.10972/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 23 references that are in the map.
- First posted on arXiv 13 Jul 2026.

## Builds on (most cited first, 9 of 9)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [DeLLMa: Decision Making Under Uncertainty with Large Language Models](https://mlsanity.com/paper/2402.02392/) (2024)
- [Look-Ahead-Bench: a Standardized Benchmark of Look-ahead Bias in Point-in-Time LLMs for Finance](https://mlsanity.com/paper/2601.13770/) (2026)
- [SSFF: Investigating LLM Predictive Capabilities for Startup Success through a Multi-Agent Framework with Enhanced Explainability and Performance](https://mlsanity.com/paper/2405.19456/) (2024)
- [GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation](https://mlsanity.com/paper/2606.22737/) (2026)
- [The Strategic Foresight of LLMs: Evidence from a Fully Prospective Venture Tournament](https://mlsanity.com/paper/2602.01684/) (2026)
- [DIALECTIC: A Multi-Agent System for Startup Evaluation](https://mlsanity.com/paper/2603.12274/) (2026)
- [Separating Diagnosis from Control: Auditable Policy Adaptation in Agent-Based Simulations with LLM-Based Diagnostics](https://mlsanity.com/paper/2603.22904/) (2026)
- [CARE: Context-Aware Ranking Evolution with Executable Scoring Programs for Budgeted Reaction Optimization](https://mlsanity.com/paper/2606.14581/) (2026)

## Abstract (arXiv)

Many evaluations of model outputs rely either on contracts checkable at evaluation time or on feedback that arrives within the operating loop. We study the complementary setting in which ground truth is delayed, censored, or private, so deterministic code cannot check correctness at scoring time and must instead issue a code-owned provisional forecast. RouteCast instantiates this regime for model-generated typed strategic routes: models propose candidate routes and structured factors; point-in-time evidence, reference classes, and deterministic transformations produce a provisional forecast-ranking; later outcomes evaluate the forecast. In a retrospective venture pilot on 21 binary-outcome cases (6 positive, 15 negative), the whole-packet RouteCast score showed preliminary retrospective discrimination (AUC 0.756, 95% CI [0.471,0.980]), while a blind LLM judge reached AUC 0.678 [0.419,0.897] and an identity-exposed LLM judge reached AUC 0.761 [0.515,0.944], consistent with recognition- or outcome-related leakage risk. A preregistered decomposition ablation on the same binary subset found that converting the identical inputs into typed staged routes was indistinguishable from the whole-packet score (Delta AUC = -0.144, 95% CI [-0.471,0.176]) and from a deterministic heuristic (Delta AUC = -0.089, 95% CI [-0.412,0.278]). The pilot establishes an auditable feasibility result and exposes failure modes; it does not establish prospective calibration, causal decision improvement, route-decomposition advantage, or cross-domain validity.