mlsanityData to 7 Oct 2026

Real-Time Trustworthiness Scoring for LLM Structured Outputs and Data Extraction

arXiv 2603.18014 ↗First posted 24 Feb 2026cs.CL cs.LGcited by 3

Not on the map. Why?

Abstract (arXiv)

Structured Outputs from current LLMs exhibit sporadic errors, hindering enterprise AI deployment. We present CONSTRUCT, a real-time uncertainty estimator that scores the trustworthiness of LLM Structured Outputs. Lower-scoring outputs are more likely to contain errors, enabling automatic prioritization of limited human review bandwidth. CONSTRUCT additionally scores the trustworthiness of each field within a Structured Output, helping reviewers quickly identify which parts of the output are incorrect. Our method is suitable for any LLM (including black-box LLM APIs without logprobs), does not require labeled training data or custom model deployment, and supports complex Structured Outputs with heterogeneous fields and nested JSON schemas. We also introduce one of the first public LLM Structured Output benchmarks with reliable ground-truth values. Over this four-dataset benchmark, CONSTRUCT detects errors in outputs from various LLMs (including Gemini 3 and GPT-5) with significantly higher precision/recall than existing techniques.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 20 of its 40 references that are in the map. First posted on arXiv 24 Feb 2026.

Builds on

20 of 40 references.

References · 20

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  4. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  5. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  6. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  7. 2025OpenAI GPT-5 System Cardcited by 910
  8. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  9. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 615
  10. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  11. 2024Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Modelscited by 431
  12. 2021Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarkscited by 388
8 more
  1. 2023Quantifying Uncertainty in Answers from any Language Model and Enhancing their Trustworthinesscited by 121
  2. 2025Uncertainty Quantification and Confidence Calibration in Large Language Models: A Surveycited by 93
  3. 2024On Verbalized Confidence Scores for LLMscited by 83
  4. 2024Lynx: An Open Source Hallucination Evaluation Modelcited by 55
  5. 2024DocETL: Agentic Query Rewriting and Evaluation for Complex Document Processingcited by 49
  6. 2019On the Calibration of Multiclass Classification with Rejectioncited by 41
  7. 2020A Benchmark for Lease Contract Reviewcited by 26
  8. 2024Dipper: Diversity in Prompts for Producing Large Language Model Ensembles in Reasoning taskscited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Beyond Logprobs: A Multi-Signal Confidence Engine for LLM-Based Document Field Extractioncited by 2
  2. 2026Hybrid AI for Explainable and Accurate Conversational Agents in eGovernmentcited by 0
  3. 2026Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Payscited by 0