mlsanityData to 7 Oct 2026

VERITYGATE: A Four-Gate Schema-Level Faithfulness Framework and Paired Benchmark for Grounded LLM Narrations over Structured Evidence

arXiv 2610.00833 ↗First posted 30 Sep 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Fluent LLM explanations may not follow the evidence from a structured system. We present VERITYGATE, a four-gate checker for declared evidence IDs, entities, numbers, and claim types. It checks a fixed schema; it does not verify every fact in the prose. At r=0 and r=1, we test 900 instances per setting (450 grounded-ungrounded pairs) with GPT-4o-mini, Llama-3.3-70B, and Claude Sonnet 4.6. Under this schema-level contract and before repair, 80.3% of mini claims and 47.9% of Sonnet claims fail. These are verifier rejection rates, not prose-hallucination rates. One repair pass raises claim survival from 19.7% to 28.0% for mini and from 52.1% to 54.3% for Sonnet. Verified claims per example change by +0.14 for mini, -0.71 for Llama, and -0.47 for Sonnet, so survival and output volume must be reported together. A second Sonnet pass gives no clear gain. At r=1, Gate 4 covers 97.0%, 98.7%, and 100% of failing claims for mini, Llama, and Sonnet. Small human studies support the rules but show gaps between schema checks and correct prose. A domain-specific GPT-4o judge test shows an order effect, so it is only a usefulness check. We release the code and data.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 14 of its 28 references that are in the map. First posted on arXiv 30 Sep 2026.

Builds on

14 of 28 references.

References · 14

  1. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  2. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  3. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  4. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  5. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  6. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 965
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  8. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  9. 2017Challenges in Data-to-Document Generationcited by 226
  10. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  11. 2019Handling Divergent Reference Texts when Evaluating Table-to-Text Generationcited by 119
  12. 2019Semantic Noise Matters for Neural Natural Language Generationcited by 64
2 more
  1. 2016Sequence-to-Sequence Generation for Spoken Dialogue via Deep Syntax Trees and Stringscited by 27
  2. 2018A Deep Ensemble Model with Slot Alignment for Sequence-to-Sequence Natural Language Generationcited by 20

Built on by

Not cited yet.