mlsanityData to 14 Oct 2026

FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generation

arXiv 2110.08559 ↗First posted 16 Oct 2021cs.CLcited by 19

Not on the map. Why?

Abstract (arXiv)

Fast and reliable evaluation metrics are key to R&D progress. While traditional natural language generation metrics are fast, they are not very reliable. Conversely, new metrics based on large pretrained language models are much more reliable, but require significant computational resources. In this paper, we propose FrugalScore, an approach to learn a fixed, low cost version of any expensive NLG metric, while retaining most of its original performance. Experiments with BERTScore and MoverScore on summarization and translation show that FrugalScore is on par with the original metrics (and sometimes better), while having several orders of magnitude less parameters and running several times faster. On average over all learned metrics, tasks, and variants, FrugalScore retains 96.8% of the performance, runs 24 times faster, and has 35 times less parameters than the original metrics. We make our trained metrics publicly available, to benefit the entire NLP community and in particular researchers and practitioners with limited resources.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 29 of its 58 references that are in the map. First posted on arXiv 16 Oct 2021.

Builds on

29 of 58 references.

References · 29

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  10. 2018Deep contextualized word representationscited by 2,290
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
17 more
  1. 2020Knowledge Distillation: A Surveycited by 1,603
  2. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,488
  3. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  5. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  7. 2017Deep Mutual Learningcited by 628
  8. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 432
  9. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  10. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  11. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  12. 2020BARThez: a Skilled Pretrained French Sequence-to-Sequence Modelcited by 40
  13. 2019Distilling Knowledge From a Deep Pose Regressor Networkcited by 34
  14. 2020NUBIA: NeUral Based Interchangeability Assessor for Text Generationcited by 34
  15. 2019Findings of the First Shared Task on Machine Translation Robustnesscited by 17
  16. 2020An Efficient Method of Training Small Models for Regression Problems with Knowledge Distillationcited by 14
  17. 2020Mark-Evaluate: Assessing Language Generation using Population Estimation Methodscited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 4; 2024: 4; 2025: 3; 2026: 1.

19 papers.

Sort:

By area:

  1. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 155
  2. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 73
  3. 2025Evaluating LLM-based Agents for Multi-Turn Conversations: A Surveycited by 57
  4. 2022BERTScore is Unfair: On Social Bias in Language Model-Based Metrics for Text Generationcited by 48
  5. 2022AraBART: a Pretrained Arabic Sequence-to-Sequence Model for Abstractive Summarizationcited by 24
  6. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  7. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  8. 2022Abstractive Meeting Summarization: A Surveycited by 9
  9. 2024xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metricscited by 7
  10. 2022EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metricscited by 4
  11. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metricscited by 4
  12. 2022DATScore: Evaluating Translation with Data Augmented Translationscited by 2
7 more
  1. 2023Improved Cross-Lingual Transfer Learning For Automatic Speech Translationcited by 2
  2. 2023Talking with Machines: A Comprehensive Survey of Emergent Dialogue Systemscited by 1
  3. 2024What Would You Ask When You First Saw a2+b2=c2? Evaluating LLM on Curiosity-Driven Questioningcited by 1
  4. 2025AraFinNews: Arabic Financial Summarisation with Domain-Adapted LLMscited by 1
  5. 2026PIPE-Cypher: Automatic Enterprise Benchmark Generation for Text-to-Cypher Systemscited by 1
  6. 2023Recent Trends in Unsupervised Summarizationcited by 0
  7. 2025ReqBrain: Task-Specific Instruction Tuning of LLMs for AI-Assisted Requirements Generationcited by 0