VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence
Not on the map. Why?
Abstract (arXiv)
Fluent LLM explanations may not follow the evidence from a structured system. We present VERITYGATE, a four-gate checker for declared evidence IDs, entities, numbers, and claim types. It checks a fixed schema; it does not verify every fact in the prose. At r=0 and r=1, we test 900 instances per setting (450 grounded-ungrounded pairs) with GPT-4o-mini, Llama-3.3-70B, and Claude Sonnet 4.6. Under this schema-level contract and before repair, 80.3% of mini claims and 47.9% of Sonnet claims fail. These are verifier rejection rates, not prose-hallucination rates. One repair pass raises claim survival from 19.7% to 28.0% for mini and from 52.1% to 54.3% for Sonnet. Verified claims per example change by +0.14 for mini, -0.71 for Llama, and -0.47 for Sonnet, so survival and output volume must be reported together. A second Sonnet pass gives no clear gain. At r=1, Gate 4 covers 97.0%, 98.7%, and 100% of failing claims for mini, Llama, and Sonnet. Small human studies support the rules but show gaps between schema checks and correct prose. A domain-specific GPT-4o judge test shows an order effect, so it is only a usefulness check. We release the code and data.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 30 Sep 2026.
Builds on
14 of 28 references.
References · 14
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
- 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
- 2023Large Language Models are not Fair Evaluatorscited by 1,050
- 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
- 2023Enabling Large Language Models to Generate Text with Citationscited by 584
- 2021Measuring Attribution in Natural Language Generation Modelscited by 253
- 2017Challenges in Data-to-Document Generationcited by 226
- 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
- 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
- 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
Built on by
Not cited yet.