LUNA: A Framework for Language Understanding and Naturalness Assessment
Not on the map. Why?
Abstract (arXiv)
The evaluation of Natural Language Generation (NLG) models has gained increased attention, urging the development of metrics that evaluate various aspects of generated text. LUNA addresses this challenge by introducing a unified interface for 20 NLG evaluation metrics. These metrics are categorized based on their reference-dependence and the type of text representation they employ, from string-based n-gram overlap to the utilization of static embeddings and pre-trained language models. The straightforward design of LUNA allows for easy extension with novel metrics, requiring just a few lines of code. LUNA offers a user-friendly tool for evaluating generated texts.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 23 of its 40 references that are in the map. First posted on arXiv 9 Jan 2024.
Builds on
23 of 40 references.
References · 23
- 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
- 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
- 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
- 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
- 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
- 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 292
- 2017Why We Need New Evaluation Metrics for NLGcited by 232
- 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
- 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
- 2020Fill in the BLANC: Human-free quality estimation of document summariescited by 80
- 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
- 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
11 more
- 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
- 2017Relevance of Unsupervised Metrics in Task-Oriented Dialogue for Evaluating Natural Language Generationcited by 61
- 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
- 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
- 2022DiscoScore: Evaluating Text Generation with BERT and Discourse Coherencecited by 35
- 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
- 2022Deconstructing NLG Evaluation: Evaluation Practices, Assumptions, and Their Implicationscited by 32
- 2020Curious Case of Language Generation Evaluation Metrics: A Cautionary Talecited by 28
- 2021A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regionscited by 17
- 2023NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklistcited by 16
- 2022Social Biases in Automatic Evaluation Metrics for NLGcited by 3
Built on by
Citing papers per year: 2024: 1.
1 paper.