mlsanityData to 7 Oct 2026

Automatic Text Evaluation through the Lens of Wasserstein Barycenters

arXiv 2108.12463 ↗First posted 27 Aug 2021cs.CL cs.AIcited by 38

Not on the map. Why?

Abstract (arXiv)

A new metric textttBaryScore to evaluate text generation based on deep contextualized embeddings e.g., BERT, Roberta, ELMo) is introduced. This metric is motivated by a new framework relying on optimal transport tools, i.e., Wasserstein distance and barycenter. By modelling the layer output of deep contextualized embeddings as a probability distribution rather than by a vector embedding; this framework provides a natural way to aggregate the different outputs through the Wasserstein space topology. In addition, it provides theoretical grounds to our metric and offers an alternative to available solutions e.g., MoverScore and BertScore). Numerical evaluation is performed on four different tasks: machine translation, summarization, data2text generation and image captioning. Our results show that textttBaryScore outperforms other BERT based metrics and exhibits more consistent behaviour in particular for text summarization.

Key facts. Cited by 38 papers in the map (data through 7 Oct 2026). Builds on 69 of its 140 references that are in the map. First posted on arXiv 27 Aug 2021.

Builds on

69 of 140 references.

References · 69

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2017Improved Training of Wasserstein GANscited by 2,223
  11. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  12. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
38 more
  1. 2015Teaching Machines to Read and Comprehendcited by 1,039
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  3. 2016SPICE: Semantic Propositional Image Caption Evaluationcited by 820
  4. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  5. 2018Computational Optimal Transportcited by 765
  6. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  7. 2017Wasserstein GANcited by 635
  8. 2019Designing and Interpreting Probes with Control Taskscited by 608
  9. 2019Text Summarization with Pretrained Encoderscited by 553
  10. 2015Optimal Transport for Domain Adaptationcited by 457
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  13. 2017Wasserstein Auto-Encoderscited by 371
  14. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
  15. 2017Challenges in Data-to-Document Generationcited by 226
  16. 2013Fast Computation of Wasserstein Barycenterscited by 216
  17. 2016Enhanced LSTM for Natural Language Inferencecited by 199
  18. 2020Extractive Summarization as Text Matchingcited by 189
  19. 2018Bottom-Up Abstractive Summarizationcited by 177
  20. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  21. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  22. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  23. 2019The Bottom-up Evolution of Representations in the Transformer: A Study with Machine Translation and Language Modeling Objectivescited by 139
  24. 2020With Little Power Comes Great Responsibilitycited by 138
  25. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  26. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  27. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  28. 2020Heterogeneous Graph Neural Networks for Extractive Document Summarizationcited by 90
  29. 2019Wasserstein Adversarial Examples via Projected Sinkhorn Iterationscited by 89
  30. 2019Better Rewards Yield Better Summaries: Learning to Summarise Without Referencescited by 76
  31. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  32. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  33. 2018Content Selection in Deep Learning Models of Summarizationcited by 64
  34. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  35. 2018Learning to Evaluate Image Captioningcited by 53
  36. 2019Affect-Driven Dialog Generationcited by 50
  37. 2017Multilevel Clustering via Wasserstein Meanscited by 48
  38. 2019Searching for Effective Neural Extractive Summarization: What Works and What's Nextcited by 38

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 18; 2023: 8; 2024: 6; 2025: 2; 2026: 1.

38 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  3. 2024Towards Cross-Tokenizer Distillation: the Universal Logit Distillation Loss for LLMscited by 61
  4. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  5. 2024Do Language Models Enjoy Their Own Stories? Prompting Large Language Models for Automatic Story Evaluationcited by 32
  6. 2022Wasserstein Iterative Networks for Barycenter Estimationcited by 23
  7. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  8. 2022Acceptability Judgements via Examining the Topology of Attention Mapscited by 19
  9. 2022MENLI: Robust Evaluation Metrics from Natural Language Inferencecited by 18
  10. 2022Unsupervised Extractive Opinion Summarization Using Sparse Codingcited by 17
  11. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  12. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
26 more
  1. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  2. 2022Towards Explainable Evaluation Metrics for Natural Language Generationcited by 15
  3. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  4. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  5. 2023OpinSummEval: Revisiting Automated Evaluation for Opinion Summarizationcited by 13
  6. 2021Beam Search with Bidirectional Strategies for Neural Response Generationcited by 12
  7. 2023Energy-Guided Continuous Entropic Barycenter Estimation for General Costscited by 12
  8. 2023Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applicationscited by 11
  9. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  10. 2022Measure Estimation in the Barycentric Coding Modelcited by 9
  11. 2022Reproducibility Issues for BERT-based Evaluation Metricscited by 9
  12. 2023Can BERT eat RuCoLA? Topological Data Analysis to Explaincited by 5
  13. 2022EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metricscited by 4
  14. 2023Transductive Learning for Textual Few-Shot Classification in API-based Embedding Modelscited by 4
  15. 2025Explainable Knowledge Graph Retrieval-Augmented Generation (KG-RAG) with KG-SMILEcited by 4
  16. 2022Beyond Mahalanobis-Based Scores for Textual OOD Detectioncited by 3
  17. 2023Toward Stronger Textual Attack Detectorscited by 3
  18. 2024PerSEval: Assessing Personalization in Text Summarizerscited by 3
  19. 2022Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Codecited by 2
  20. 2023A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classificationcited by 1
  21. 2024LUNA: A Framework for Language Understanding and Naturalness Assessmentcited by 1
  22. 2024Risk-Sensitive Diffusion: Robustly Optimizing Diffusion Models with Noisy Samplescited by 1
  23. 2025Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluationcited by 1
  24. 2022What are the best systems? New perspectives on NLP Benchmarkingcited by 0
  25. 2022Connecting adversarial attacks and optimal transport for domain adaptationcited by 0
  26. 2026Grammatical Error Correction Evaluation by Optimally Transporting Edit Representationcited by 0