mlsanityData to 7 Oct 2026

AlignScore: Evaluating Factual Consistency with a Unified Alignment Function

arXiv 2305.16739 ↗First posted 26 May 2023cs.CLcited by 284

Not on the map. Why?

Cited by 35 in Retrieval-augmented generation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Many text generation applications require the generated text to be factually consistent with input information. Automatic evaluation of factual consistency is challenging. Previous work has developed various metrics that often depend on specific functions, such as natural language inference (NLI) or question answering (QA), trained on limited data. Those metrics thus can hardly assess diverse factual inconsistencies (e.g., contradictions, hallucinations) that occur in varying inputs/outputs (e.g., sentences, documents) from different tasks. In this paper, we propose AlignScore, a new holistic metric that applies to a variety of factual inconsistency scenarios as above. AlignScore is based on a general function of information alignment between two arbitrary text pieces. Crucially, we develop a unified training framework of the alignment function by integrating a large diversity of data sources, resulting in 4.7M training examples from 7 well-established tasks (NLI, QA, paraphrasing, fact verification, information retrieval, semantic similarity, and summarization). We conduct extensive experiments on large-scale benchmarks including 22 evaluation datasets, where 19 of the datasets were never seen in the alignment training. AlignScore achieves substantial improvement over a wide range of previous metrics. Moreover, AlignScore (355M parameters) matches or even outperforms metrics based on ChatGPT and GPT-4 that are orders of magnitude larger.

Key facts. Cited by 284 papers in the map (data through 7 Oct 2026). Builds on 57 of its 83 references that are in the map. First posted on arXiv 26 May 2023.

Builds on

57 of 83 references.

References · 57

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016Pointer Sentinel Mixture Modelscited by 2,992
  7. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  10. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  11. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  12. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
38 more
  1. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  2. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  3. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  6. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  7. 2017RACE: Large-scale ReAding Comprehension Dataset From Examinationscited by 851
  8. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  9. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  13. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  14. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  15. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 395
  16. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  17. 2023GPTScore: Evaluate as You Desirecited by 357
  18. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  19. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  20. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  21. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  22. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  23. 2022UnifiedSKG: Unifying and Multi-Tasking Structured Knowledge Grounding with Text-to-Text Language Modelscited by 231
  24. 2021Get Your Vitamin C! Robust Fact Verification with Contrastive Evidencecited by 222
  25. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  26. 2018Marian: Fast Neural Machine Translation in C++cited by 202
  27. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  28. 2017Faithful to the Original: Fact Aware Neural Abstractive Summarizationcited by 141
  29. 2021Q2: Evaluating Factual Consistency in Knowledge-Grounded Dialogues via Question Generation and Question Answeringcited by 131
  30. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  31. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  32. 2018Combining Fact Extraction and Verification with Neural Semantic Matching Networkscited by 120
  33. 2018Transforming Question Answering Datasets Into Natural Language Inference Datasetscited by 99
  34. 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
  35. 2020GO FIGURE: A Meta Evaluation of Factuality in Summarizationcited by 75
  36. 2021Evaluating Attribution in Dialogue Systems: The BEGIN Benchmarkcited by 75
  37. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  38. 2020What Have We Achieved on Text Summarization?cited by 65

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 12; 2024: 98; 2025: 95; 2026: 79.

284 papers.

Sort:

By area:

  1. 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
  2. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  3. 2023Large Language Model Alignment: A Surveycited by 241
  4. 2023HalluciDoctor: Mitigating Hallucinatory Toxicity in Visual Instruction Datacited by 124
  5. 2024Benchmarking Uncertainty Quantification Methods for Large Language Models with LM-Polygraphcited by 105
  6. 2024Trust or Escalate: LLM Judges with Provable Guarantees for Human Agreementcited by 99
  7. 2024Summary of a Haystack: A Challenge to Long-Context LLMs and RAG Systemscited by 87
  8. 2024Fast Adversarial Attacks on Language Models In One GPU Minutecited by 85
  9. 2024A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendationscited by 84
  10. 2024Large Language Model (LLM) for Telecommunications: A Comprehensive Survey on Principles, Key Techniques, and Opportunitiescited by 83
  11. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  12. 2024Navigating the Risks: A Survey of Security, Privacy, and Ethics Threats in LLM-Based Agentscited by 63
38 more
  1. 2025Evaluating Step-by-step Reasoning Traces: A Surveycited by 62
  2. 2024Molecular Facts: Desiderata for Decontextualization in LLM Fact Verificationcited by 42
  3. 2024AdaCAD: Adaptively Decoding to Balance Conflicts between Contextual and Parametric Knowledgecited by 35
  4. 2023Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinationscited by 34
  5. 2024Decomposition Dilemmas: Does Claim Decomposition Boost or Burden Fact-Checking Performance?cited by 34
  6. 2023Creating Trustworthy LLMs: Dealing with Hallucinations in Healthcare AIcited by 33
  7. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  8. 2023Aligning Large Language Models with Human: A Surveycited by 31
  9. 2025ICR Probe: Tracking Hidden State Dynamics for Reliable Hallucination Detection in LLMscited by 31
  10. 2024Beyond Aesthetics: Cultural Competence in Text-to-Image Modelscited by 30
  11. 2025Uncertainty Quantification for LLMs through Minimum Bayes Risk: Bridging Confidence and Consistencycited by 30
  12. 2024Beyond correlation: The Impact of Human Uncertainty in Measuring the Effectiveness of Automatic Evaluation and LLM-as-a-Judgecited by 29
  13. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  14. 2024Learning to Refine with Fine-Grained Natural Language Feedbackcited by 27
  15. 2024RefChecker: Reference-based Fine-grained Hallucination Checker and Benchmark for Large Language Modelscited by 24
  16. 2024FaithBench: A Diverse Hallucination Benchmark for Summarization by Modern LLMscited by 24
  17. 2024RAG-RLRC-LaySum at BioLaySumm: Integrating Retrieval-Augmented Generation and Readability Control for Layman Summarization of Biomedical Textscited by 23
  18. 2024Claim Verification in the Age of Large Language Models: A Surveycited by 23
  19. 2024Proactive Agents for Multi-Turn Text-to-Image Generation Under Uncertaintycited by 23
  20. 2024On Positional Bias of Faithfulness for Long-form Summarizationcited by 22
  21. 2025The Illusion of Progress: Re-evaluating Hallucination Detection in LLMscited by 22
  22. 2024On the Essence and Prospect: An Investigation of Alignment Approaches for Big Modelscited by 21
  23. 2024TimeChara: Evaluating Point-in-Time Character Hallucination of Role-Playing Large Language Modelscited by 21
  24. 2025PeerQA: A Scientific Question Answering Dataset from Peer Reviewscited by 21
  25. 2025Revisiting Uncertainty Quantification Evaluation in Language Models: Spurious Interactions with Response Length Bias Resultscited by 21
  26. 2024Hallucination Detection and Hallucination Mitigation: An Investigationcited by 20
  27. 2024Are LLMs Better than Reported? Detecting Label Errors and Mitigating Their Effect on Model Performancecited by 20
  28. 2024Reading Subtext: Evaluating Large Language Models on Short Story Summarization with Writerscited by 19
  29. 2024Automatic Metrics in Natural Language Generation: A Survey of Current Evaluation Practicescited by 19
  30. 2024Overview of the First Shared Task on Clinical Text Generation: RRG24 and "Discharge Me!"cited by 19
  31. 2024Retrieval-Augmented Generation with Estimation of Source Reliabilitycited by 19
  32. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 19
  33. 2024FactPICO: Factuality Evaluation for Plain Language Summarization of Medical Evidencecited by 18
  34. 2024FoRAG: Factuality-optimized Retrieval Augmented Generation for Web-enhanced Long-form Question Answeringcited by 18
  35. 2024Overview of the BioLaySumm 2024 Shared Task on the Lay Summarization of Biomedical Research Articlescited by 18
  36. 2024ECon: On the Detection and Resolution of Evidence Conflictscited by 18
  37. 2025Token-Level Density-Based Uncertainty Quantification Methods for Eliciting Truthfulness of Large Language Modelscited by 18
  38. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17

50 most cited of 284 All 284 (JSON)