mlsanityData to 7 Oct 2026

Measuring Black-Box Confidence via Reasoning Trajectories: Geometry, Coverage, and Verbalization

arXiv 2605.06308 ↗First posted 7 May 2026cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Reliable confidence estimation enables safe deployment of chain-of-thought (CoT) reasoning through text-only APIs. Yet the dominant black-box baseline, self-consistency over K samples, is linearly expensive and ignores the geometry of the trace. We propose a black-box trajectory-confidence score: we embed a CoT as a sliding-window trajectory and measure its convergence to external answer anchors with a one-parameter softmax. The method needs no logits, hidden states, or supervised calibrators. Across six (benchmark, reasoner) settings on MedQA-USMLE, GPQA Diamond, and MMLU-Pro with Gemini 3.1 Pro and Claude Sonnet 4.6, fusing this score with coverage and verbalized-confidence channels at K=4 yields Pareto improvements over self-consistency at K=8 in 6/6 settings (median AUC 0.78 vs 0.71, deltaAUC=+0.075). A fixed-pick control (+0.060) and E5 cross-embedder replication rule out answer switching and single-vendor artifacts. Geometry peaks in the penultimate window across benchmarks and reasoners, and inverts at the terminal window on GPQA Diamond. Three unscaffolded regimes separate black-box confidence into a judge-mediated Coverage prior (C), within-trace Geometry (G), and a conditional Verbalization channel (V). Across 18 benchmark x reasoner x proposer settings, C and G provide independent signal in 18/18 and 16/18, while V contributes residual signal in 6/18. Swapping the judge from GPT-5-mini to Claude Sonnet 4.6 leaves G-only AUC unchanged (|delta|<=0.013) and shifts C-only AUC by at most +/-0.02 (kappa=0.82). Fusion beats the best single channel in 17/18 settings (median AUC 0.78, max 0.92).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 25 of its 29 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

25 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Language Models (Mostly) Know What They Know11 Jul

Other references · 24

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  4. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  5. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  6. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  7. 2019How Can We Know What Language Models Know?cited by 953
  8. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  9. 2023Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generationcited by 795
  10. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  11. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  12. 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
12 more
  1. 2023The Internal State of an LLM Knows When It's Lyingcited by 644
  2. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  3. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  4. 2023Generating with Confidence: Uncertainty Quantification for Black-box Large Language Modelscited by 286
  5. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  6. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  7. 2023Selectively Answering Ambiguous Questionscited by 63
  8. 2024Large Language Model Confidence Estimation via Black-Box Accesscited by 14
  9. 2025Towards Agents That Know When They Don't Know: Uncertainty as a Control Signal for Structured Reasoningcited by 10
  10. 2025Query, Don't Train: Privacy-Preserving Tabular Prediction from EHR Data via SQL Queriescited by 5
  11. 2025Towards Label-Free Biological Reasoning Synthetic Dataset Creation via Uncertainty Filteringcited by 3
  12. 2026MechPert: Mechanistic Consensus as an Inductive Bias for Unseen Perturbation Predictioncited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Post-Training Shifts Confidence: A Three-Stage Analysis of How SFT, RL, and OPD Shape CoT Calibration
  2. 2026Confidence Geometry Reveals Trace-Level Correctness in Large Language Model Reasoning
  3. 2026Diagnosing Multi-step Reasoning Failures in Black-box LLMs via Stepwise Confidence Attribution
  4. 2026Early Stopping for Large Reasoning Models via Confidence Dynamics
  5. 2025Verbalized Confidence Triggers Self-Verification: Emergent Behavior Without Explicit Reasoning Supervision
  6. 2026When Confidence Rises Too Early: Detecting Shortcut Reasoning via Premature Answer Commitment
  7. 2026ThinkRouter: Efficient Reasoning via Routing Thinking between Latent and Discrete Spaces
  8. 2025Temporalizing Confidence: Evaluation of Chain-of-Thought Reasoning with Signal Temporal Logic