mlsanityData to 7 Oct 2026

InfoLM: A New Metric to Evaluate Summarization & Data2Text Generation

arXiv 2112.01589 ↗First posted 2 Dec 2021cs.CL cs.AIcited by 33

Not on the map. Why?

Abstract (arXiv)

Assessing the quality of natural language generation systems through human annotation is very expensive. Additionally, human annotation campaigns are time-consuming and include non-reusable human labour. In practice, researchers rely on automatic metrics as a proxy of quality. In the last decade, many string-based metrics (e.g., BLEU) have been introduced. However, such metrics usually rely on exact matches and thus, do not robustly handle synonyms. In this paper, we introduce InfoLM a family of untrained metrics that can be viewed as a string-based metric that addresses the aforementioned flaws thanks to a pre-trained masked language model. This family of metrics also makes use of information measures allowing the adaptation of InfoLM to various evaluation criteria. Using direct assessment, we demonstrate that InfoLM achieves statistically significant improvement and over 10 points of correlation gains in many configurations on both summarization and data2text generation.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Builds on 49 of its 112 references that are in the map. First posted on arXiv 2 Dec 2021.

Builds on

49 of 112 references.

References · 49

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2017On Calibration of Modern Neural Networkscited by 4,860
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2018Deep contextualized word representationscited by 2,285
  10. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  11. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  12. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
37 more
  1. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  2. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  3. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  5. 2019Text Summarization with Pretrained Encoderscited by 553
  6. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  7. 2020Extractive Summarization as Text Matchingcited by 189
  8. 2018Bottom-Up Abstractive Summarizationcited by 177
  9. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  10. 2020Re-evaluating Evaluation in Text Summarizationcited by 144
  11. 2015Semantically Conditioned LSTM-based Natural Language Generation for Spoken Dialogue Systemscited by 134
  12. 2018Ranking Sentences for Extractive Summarization with Reinforcement Learningcited by 128
  13. 2016R'enyi Divergence Variational Inferencecited by 104
  14. 2020Heterogeneous Graph Neural Networks for Extractive Document Summarizationcited by 90
  15. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  16. 2018ROUGE 2.0: Updated and Improved Measures for Evaluation of Summarization Taskscited by 66
  17. 2018Content Selection in Deep Learning Models of Summarizationcited by 64
  18. 2018Neural Document Summarization by Jointly Learning to Score and Select Sentencescited by 56
  19. 2019Affect-Driven Dialog Generationcited by 50
  20. 2019Searching for Effective Neural Extractive Summarization: What Works and What's Nextcited by 38
  21. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  22. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  23. 2020Hierarchical Pre-training for Sequence Labelling in Spoken Dialogcited by 36
  24. 2019Calibration, Entropy Rates, and Memory in Language Modelscited by 32
  25. 2020Guiding attention in Sequence-to-sequence models for Dialogue Act predictioncited by 26
  26. 2020Example-Driven Intent Prediction with Observerscited by 26
  27. 2020When OT meets MoM: Robust estimation of Wasserstein Distancecited by 23
  28. 2015deltaBLEU: A Discriminative Metric for Generation Tasks with Intrinsically Diverse Targetscited by 21
  29. 2018Concatenated Power Mean Word Embeddings as Universal Cross-Lingual Sentence Representationscited by 20
  30. 2020Heavy-tailed Representations, Text Polarity Classification & Data Augmentationcited by 20
  31. 2021Adversarial Robustness via Fisher-Rao Regularizationcited by 19
  32. 2020The importance of fillers for text representations of speech transcriptscited by 15
  33. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  34. 2019From the Token to the Review: A Hierarchical Multimodal approach to Opinion Miningcited by 14
  35. 2018Alpha-Beta Divergence For Variational Inferencecited by 12
  36. 2021Beam Search with Bidirectional Strategies for Neural Response Generationcited by 12
  37. 2020Learning by Semantic Similarity Makes Abstractive Summarization Bettercited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 8; 2024: 9; 2025: 2; 2026: 4.

33 papers.

Sort:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  2. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  3. 2023CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generationcited by 65
  4. 2023Language Models Hallucinate, but May Excel at Fact Verificationcited by 35
  5. 2024Inherent Trade-Offs between Diversity and Stability in Multi-Task Benchmarkscited by 26
  6. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  7. 2023MuseChat: A Conversational Music Recommendation System for Videoscited by 21
  8. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  9. 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
  10. 2022Vote'n'Rank: Revision of Benchmarking with Social Choice Theorycited by 15
  11. 2023Unsupervised Layer-wise Score Aggregation for Textual OOD Detectioncited by 15
  12. 2024Hyperparameter-Free Approach for Faster Minimum Bayes Risk Decodingcited by 15
21 more
  1. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  2. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  3. 2024ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilitiescited by 13
  4. 2024From Variability to Stability: Advancing RecSys Benchmarking Practicescited by 12
  5. 2023Revisiting Instruction Fine-tuned Model Evaluation to Guide Industrial Applicationscited by 11
  6. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  7. 2023DecompEval: Evaluating Generated Texts as Unsupervised Decomposed Question Answeringcited by 7
  8. 2024GraphNarrator: Generating Textual Explanations for Graph Neural Networkscited by 7
  9. 2022WikiDes: A Wikipedia-Based Dataset for Generating Short Descriptions from Paragraphscited by 6
  10. 2024A Guide to Similarity Measurescited by 4
  11. 2022Beyond Mahalanobis-Based Scores for Textual OOD Detectioncited by 3
  12. 2024PerSEval: Assessing Personalization in Text Summarizerscited by 3
  13. 2026BERT-as-a-Judge: A Robust Alternative to Lexical Methods for Efficient Reference-Based LLM Evaluationcited by 3
  14. 2022Repro: An Open-Source Library for Improving the Reproducibility and Usability of Publicly Available Research Codecited by 2
  15. 2025Fact-Controlled Diagnosis of Hallucinations in Medical Text Summarizationcited by 2
  16. 2025Metritocracy: Representative Metrics for Lite Benchmarkscited by 2
  17. 2024LUNA: A Framework for Language Understanding and Naturalness Assessmentcited by 1
  18. 2026coTherapist: A Behavior-Aligned Small Language Model to Support Mental Healthcare Expertscited by 1
  19. 2022What are the best systems? New perspectives on NLP Benchmarkingcited by 0
  20. 2026Towards a Linguistic Evaluation of Narratives: A Quantitative Stylistic Frameworkcited by 0
  21. 2026An Information-Geometric Justification for Composite Coherence in Event-Based Narrative Extractioncited by 0