mlsanityData to 7 Oct 2026

Measuring the Measuring Tools: An Automatic Evaluation of Semantic Metrics for Text Corpora

arXiv 2211.16259 ↗First posted 29 Nov 2022cs.CLcited by 11

Not on the map. Why?

Abstract (arXiv)

The ability to compare the semantic similarity between text corpora is important in a variety of natural language processing applications. However, standard methods for evaluating these metrics have yet to be established. We propose a set of automatic and interpretable measures for assessing the characteristics of corpus-level semantic similarity metrics, allowing sensible comparison of their behavior. We demonstrate the effectiveness of our evaluation measures in capturing fundamental characteristics by evaluating them on a collection of classical and state-of-the-art metrics. Our measures revealed that recently-developed metrics are becoming better in identifying semantic distributional mismatch while classical metrics are more sensitive to perturbations in the surface text levels.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 16 of its 33 references that are in the map. First posted on arXiv 29 Nov 2022.

Builds on

16 of 33 references.

References · 16

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  3. 2019Improved Precision and Recall Metric for Assessing Generative Modelscited by 925
  4. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  5. 2020Reliable Fidelity and Diversity Metrics for Generative Modelscited by 425
  6. 2020Efficient Intent Detection with Dual Sentence Encoderscited by 419
  7. 2018Assessing Generative Models via Precision and Recallcited by 401
  8. 2019An Evaluation Dataset for Intent Classification and Out-of-Scope Predictioncited by 378
  9. 2017Are GANs Created Equal? A Large-Scale Studycited by 267
  10. 2016Revisiting Classifier Two-Sample Testscited by 258
  11. 2017Why We Need New Evaluation Metrics for NLGcited by 232
  12. 2017Learning Discourse-level Diversity for Neural Dialog Models using Conditional Variational Autoencoderscited by 222
4 more
  1. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  2. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  3. 2022GEMv2: Multilingual NLG Benchmarking in a Single Line of Codecited by 36
  4. 2021Automatic Construction of Evaluation Suites for Natural Language Generation Datasetscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 3; 2024: 5; 2025: 1; 2026: 1.

11 papers.

  1. 2023Harnessing large-language models to generate private synthetic textcited by 65
  2. 2022MAUVE Scores for Generative Models: Theory and Practicecited by 34
  3. 2024Leveraging Machine Learning to Identify Gendered Stereotypes and Body Image Concerns on Diet and Fitness Online Forumscited by 3
  4. 2024Controlled Automatic Task-Specific Synthetic Data Generation for Hallucination Detectioncited by 3
  5. 2024Improving and Assessing the Fidelity of Large Language Models Alignment to Online Communitiescited by 2
  6. 2023Reliable and Interpretable Drift Detection in Streams of Short Textscited by 1
  7. 2024Large Language Models Help Reveal Unhealthy Diet and Body Concerns in Online Eating Disorders Communitiescited by 1
  8. 2025LaajMeter: A Framework for LaaJ Evaluationcited by 1
  9. 2023A ModelOps-based Framework for Intelligent Medical Knowledge Extractioncited by 0
  10. 2024Depth F1: Improving Evaluation of Cross-Domain Text Classification by Measuring Semantic Generalizabilitycited by 0
  11. 2026Leveraging Speech Acts for Low-Data and Cross-Domain Conversation Derailment Forecastingcited by 0