mlsanityData to 7 Oct 2026

BARTScore: Evaluating Generated Text as Text Generation

arXiv 2106.11520 ↗First posted 22 Jun 2021cs.CLcited by 719

Not on the map. Why?

Cited by 28 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.

Key facts. Cited by 719 papers in the map (data through 7 Oct 2026). Builds on 48 of its 78 references that are in the map. First posted on arXiv 22 Jun 2021.

Builds on

48 of 78 references.

References · 48

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  7. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  8. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  12. 2015Teaching Machines to Read and Comprehendcited by 1,039
36 more
  1. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  2. 2019How Can We Know What Language Models Know?cited by 953
  3. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  5. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2019Text Summarization with Pretrained Encoderscited by 553
  10. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  11. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  12. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  13. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  14. 2015Semi-supervised Sequence Learningcited by 198
  15. 2021How Many Data Points is a Prompt Worth?cited by 197
  16. 2020Extractive Summarization as Text Matchingcited by 189
  17. 2018Bottom-Up Abstractive Summarizationcited by 177
  18. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  19. 2020Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metricscited by 155
  20. 2019Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answeringcited by 147
  21. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  22. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  23. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  24. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  25. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  26. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  27. 2020SciREX: A Challenge Dataset for Document-Level Information Extractioncited by 111
  28. 2020Heterogeneous Graph Neural Networks for Extractive Document Summarizationcited by 90
  29. 2021Can We Automate Scientific Reviewing?cited by 63
  30. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  31. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  32. 2018BanditSum: Extractive Summarization as a Contextual Banditcited by 46
  33. 2017Learning Language Representations for Typology Predictioncited by 40
  34. 2019Pretraining-Based Natural Language Generation for Text Summarizationcited by 40
  35. 2019Searching for Effective Neural Extractive Summarization: What Works and What's Nextcited by 38
  36. 2020Metrics also Disagree in the Low Scoring Range: Revisiting Summarization Evaluation Metricscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 13; 2022: 110; 2023: 177; 2024: 213; 2025: 142; 2026: 64.

719 papers.

Sort:

By area:

  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,050
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2023SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Modelscited by 787
  6. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  7. 2023Prometheus: Inducing Fine-grained Evaluation Capability in Language Modelscited by 558
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
  9. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  10. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  11. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  12. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 420
38 more
  1. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 408
  2. 2023GPTScore: Evaluate as You Desirecited by 357
  3. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  4. 2023Check Your Facts and Try Again: Improving Large Language Models with External Knowledge and Automated Feedbackcited by 351
  5. 2023JudgeLM: Fine-tuned Large Language Models are Scalable Judgescited by 351
  6. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 314
  7. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  8. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  9. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  10. 2023PIXIU: A Large Language Model, Instruction Data and Evaluation Benchmark for Financecited by 217
  11. 2022MuRAG: Multimodal Retrieval-Augmented Generator for Open Question Answering over Images and Textcited by 216
  12. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  13. 2023Data-centric Artificial Intelligence: A Surveycited by 202
  14. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  15. 2022ROSCOE: A Suite of Metrics for Scoring Step-by-Step Reasoningcited by 175
  16. 2023SciMON: Scientific Inspiration Machines Optimized for Noveltycited by 174
  17. 2023Benchmarking Foundation Models with Language-Model-as-an-Examinercited by 172
  18. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  19. 2023Generative Judge for Evaluating Alignmentcited by 163
  20. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  21. 2023Active Prompting with Chain-of-Thought for Large Language Modelscited by 154
  22. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  23. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  24. 2024FinBen: A Holistic Financial Benchmark for Large Language Modelscited by 140
  25. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  26. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  27. 2022Natural Language to Code Translation with Executioncited by 122
  28. 2023Automatic Prompt Augmentation and Selection with Chain-of-Thought from Labeled Datacited by 119
  29. 2023Opportunities and Challenges for ChatGPT and Large Language Models in Biomedicine and Healthcited by 118
  30. 2024A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directionscited by 115
  31. 2023A Critical Evaluation of Evaluations for Long-form Question Answeringcited by 111
  32. 2021WebQA: Multihop and Multimodal QAcited by 109
  33. 2024Benchmarking Large Language Models on Answering and Explaining Challenging Medical Questionscited by 106
  34. 2022PromptCap: Prompt-Guided Task-Aware Image Captioningcited by 104
  35. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  36. 2024Bias and Unfairness in Information Retrieval Systems: New Challenges in the LLM Eracited by 97
  37. 2023LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarizationcited by 96
  38. 2024From Matching to Generation: A Survey on Generative Information Retrievalcited by 89

50 most cited of 719 All 719 (JSON)