# FECT: Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts

arXiv: https://arxiv.org/abs/2508.00889 · HTML: https://mlsanity.com/paper/2508.00889/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 15 of its 45 references that are in the map.
- First posted on arXiv 26 Jul 2025.

## Builds on (most cited first, 15 of 15)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agents](https://mlsanity.com/paper/2504.12516/) (2025)
- [Prometheus: Inducing Fine-grained Evaluation Capability in Language Models](https://mlsanity.com/paper/2310.08491/) (2023)
- [Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamics](https://mlsanity.com/paper/2009.10795/) (2020)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?](https://mlsanity.com/paper/2503.24235/) (2025)
- [OpenFactCheck: A Unified Framework for Factuality Evaluation of LLMs](https://mlsanity.com/paper/2408.11832/) (2024)
- [HalluDial: A Large-Scale Benchmark for Automatic Dialogue-Level Hallucination Evaluation](https://mlsanity.com/paper/2406.07070/) (2024)
- [FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMs](https://mlsanity.com/paper/2410.13210/) (2024)
- [Analyzing and Evaluating Faithfulness in Dialogue Summarization](https://mlsanity.com/paper/2210.11777/) (2022)
- [A Taxonomy of Rater Disagreements: Surveying Challenges & Opportunities from the Perspective of Annotating Online Toxicity](https://mlsanity.com/paper/2311.04345/) (2023)
- [Increasing faithfulness in human-human dialog summarization with Spoken Language Understanding tasks](https://mlsanity.com/paper/2409.10070/) (2024)

## Abstract (arXiv)

Large language models (LLMs) are known to hallucinate, producing natural language outputs that are not grounded in the input, reference materials, or real-world knowledge. In enterprise applications where AI features support business decisions, such hallucinations can be particularly detrimental. LLMs that analyze and summarize contact center conversations introduce a unique set of challenges for factuality evaluation, because ground-truth labels often do not exist for analytical interpretations about sentiments captured in the conversation and root causes of the business problems. To remedy this, we first introduce a 3D -- Decompose, Decouple, Detach -- paradigm in the human annotation guideline and the LLM-judges' prompt to ground the factuality labels in linguistically-informed evaluation criteria. We then introduce FECT, a novel benchmark dataset for Factuality Evaluation of Interpretive AI-Generated Claims in Contact Center Conversation Transcripts, labeled under our 3D paradigm. Lastly, we report our findings from aligning LLM-judges on the 3D paradigm. Overall, our findings contribute a new approach for automatically evaluating the factuality of outputs generated by an AI system for analyzing contact center conversations.