mlsanityData to 7 Oct 2026

Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answering

arXiv 2104.08202 ↗First posted 16 Apr 2021cs.CLcited by 131

Not on the map. Why?

Abstract (arXiv)

Neural knowledge-grounded generative models for dialogue often produce content that is factually inconsistent with the knowledge they rely on, making them unreliable and limiting their applicability. Inspired by recent work on evaluating factual consistency in abstractive summarization, we propose an automatic evaluation metric for factual consistency in knowledge-grounded dialogue using automatic question generation and question answering. Our metric, denoted Q2, compares answer spans using natural language inference (NLI), instead of token-based matching as done in previous work. To foster proper evaluation, we curate a novel dataset of dialogue system outputs for the Wizard-of-Wikipedia dataset, manually annotated for factual consistency. We perform a thorough meta-evaluation of Q2 against other metrics using this dataset and two others, where it consistently shows higher correlation with human judgements.

Key facts. Cited by 131 papers in the map (data through 7 Oct 2026). Builds on 38 of its 57 references that are in the map. First posted on arXiv 16 Apr 2021.

Builds on

38 of 57 references.

References · 38

  1. 2017Attention Is All You Needcited by 59,932
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  9. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2018Hierarchical Neural Story Generationcited by 1,182
  12. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
26 more
  1. 2018Personalizing Dialogue Agents: I have a dog, do you have pets too?cited by 820
  2. 2020Recipes for building an open-domain chatbotcited by 593
  3. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  4. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  5. 2020Towards a Human-like Open-Domain Chatbotcited by 467
  6. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  7. 2016How NOT To Evaluate Your Dialogue System: An Empirical Study of Unsupervised Evaluation Metrics for Dialogue Response Generationcited by 356
  8. 2018AllenNLP: A Deep Semantic Natural Language Processing Platformcited by 347
  9. 2017Six Challenges for Neural Machine Translationcited by 334
  10. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  11. 2020Evaluation of Text Generation: A Surveycited by 275
  12. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  13. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  14. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  15. 2017ParlAI: A Dialog Research Software Platformcited by 139
  16. 2018Dialogue Natural Language Inferencecited by 135
  17. 2020On Exposure Bias, Hallucination and Domain Shift in Neural Machine Translationcited by 127
  18. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  19. 2017Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responsescited by 98
  20. 2018Automated Fact Checking: Task formulations, methods and future directionscited by 96
  21. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  22. 2019Domain Robustness in Neural Machine Translationcited by 64
  23. 2019Question Answering as an Automatic Evaluation Metric for News Article Summarizationcited by 63
  24. 2019The Dialogue Dodecathlon: Open-Domain Knowledge and Image Grounded Conversational Agentscited by 45
  25. 2020Designing Precise and Robust Dialogue Response Evaluatorscited by 37
  26. 2016A Fast Unified Model for Parsing and Sentence Understandingcited by 34

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 29; 2023: 41; 2024: 40; 2025: 8; 2026: 8.

131 papers.

Sort:

By area:

  1. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  2. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2024Large Language Models: A Surveycited by 506
  5. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  6. 2022RARR: Researching and Revising What Language Models Say, Using Language Modelscited by 286
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  8. 2021Measuring Attribution in Natural Language Generation Modelscited by 253
  9. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  10. 2024LLMs Know More Than They Show: On the Intrinsic Representation of LLM Hallucinationscited by 230
  11. 2024Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations?cited by 227
  12. 2023LM vs LM: Detecting Factual Errors via Cross Examinationcited by 155
38 more
  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2023Evaluating Correctness and Faithfulness of Instruction-Following Models for Question Answeringcited by 150
  3. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  4. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  5. 2023What You See is What You Read? Improving Text-Image Alignment Evaluationcited by 105
  6. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  7. 2023LLMs as Narcissistic Evaluators: When Ego Inflates Evaluation Scorescited by 88
  8. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  9. 2023Breaking Common Sense: WHOOPS! A Vision-and-Language Benchmark of Synthetic and Compositional Imagescited by 79
  10. 2023Model Merging by Uncertainty-Based Gradient Matchingcited by 77
  11. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  12. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  13. 2023Efficient Benchmarking of Language Modelscited by 71
  14. 2023WikiChat: Stopping the Hallucination of Large Language Model Chatbots by Few-Shot Grounding on Wikipediacited by 67
  15. 2022Tomayto, Tomahto. Beyond Token-level Answer Equivalence for Question Answering Evaluationcited by 66
  16. 2022All You May Need for VQA are Image Captionscited by 65
  17. 2024Risk Taxonomy, Mitigation, and Assessment Benchmarks of Large Language Model Systemscited by 63
  18. 2021DialFact: A Benchmark for Fact-Checking in Dialoguecited by 62
  19. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  20. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  21. 2022Conditional Generation with a Question-Answering Blueprintcited by 44
  22. 2022mFACE: Multilingual Summarization with Factual Consistency Evaluationcited by 42
  23. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  24. 2023Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generatorscited by 42
  25. 2023Opportunities and Challenges in Neural Dialog Tutoringcited by 33
  26. 2024Genie: Achieving Human Parity in Content-Grounded Datasets Generationcited by 31
  27. 2024Measuring and Reducing LLM Hallucination without Gold-Standard Answerscited by 31
  28. 2024K-QA: A Real-World Medical Q&A Benchmarkcited by 30
  29. 2021Rome was built in 1776: A Case Study on Factual Correctness in Knowledge-Grounded Response Generationcited by 29
  30. 2022Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methodscited by 27
  31. 2023The Dangers of trusting Stochastic Parrots: Faithfulness and Trust in Open-domain Conversational Question Answeringcited by 26
  32. 2024KnowHalu: Hallucination Detection via Multi-Form Knowledge Based Factual Checkingcited by 26
  33. 2024I Don't Know: Explicit Modeling of Uncertainty with an [IDK] Tokencited by 26
  34. 2024Book2Dial: Generating Teacher-Student Interactions from Textbooks for Cost-Effective Development of Educational Chatbotscited by 25
  35. 2024RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Modelscited by 24
  36. 2024Deductive Closure Training of Language Models for Coherence, Accuracy, and Updatabilitycited by 22
  37. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  38. 2023Improving Factual Consistency for Knowledge-Grounded Dialogue Systems via Knowledge Enhancement and Alignmentcited by 19

50 most cited of 131 All 131 (JSON)