mlsanityData to 7 Oct 2026

EffEval: A Comprehensive Evaluation of Efficiency for MT Evaluation Metrics

arXiv 2209.09593 ↗First posted 20 Sep 2022cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Efficiency is a key property to foster inclusiveness and reduce environmental costs, especially in an era of LLMs. In this work, we provide a comprehensive evaluation of efficiency for MT evaluation metrics. Our approach involves replacing computation-intensive transformers with lighter alternatives and employing linear and quadratic approximations for alignment algorithms on top of LLM representations. We evaluate six (reference-free and reference-based) metrics across three MT datasets and examine 16 lightweight transformers. In addition, we look into the training efficiency of metrics like COMET by utilizing adapters. Our results indicate that (a) TinyBERT provides the optimal balance between quality and efficiency, (b) CPU speed-ups are more substantial than those on GPU; (c) WMD approximations yield no efficiency gains while reducing quality and (d) adapters enhance training efficiency (regarding backward pass speed and memory requirements) as well as, in some cases, metric quality. These findings can help to strike a balance between evaluation speed and quality, which is essential for effective NLG systems. Furthermore, our research contributes to the ongoing efforts to optimize NLG evaluation metrics with minimal impact on performance. To our knowledge, ours is the most comprehensive analysis of different aspects of efficiency for MT metrics conducted so far.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 42 of its 66 references that are in the map. First posted on arXiv 20 Sep 2022.

Builds on

42 of 66 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  11. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  12. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
30 more
  1. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  2. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  3. 2020MPNet: Masked and Permuted Pre-training for Language Understandingcited by 973
  4. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  5. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2020Making Monolingual Sentence Embeddings Multilingual using Knowledge Distillationcited by 591
  8. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  9. 2018Universal Sentence Encodercited by 485
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  11. 2020MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfercited by 443
  12. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  13. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  14. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
  15. 2020Evaluation of Text Generation: A Surveycited by 275
  16. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  17. 2019WikiMatrix: Mining 135M Parallel Sentences in 1620 Language Pairs from Wikipediacited by 221
  18. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  19. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  20. 2020Pre-trained Summarization Distillationcited by 70
  21. 2021Scientific Credibility of Machine Translation Research: A Meta-Evaluation of 769 Paperscited by 70
  22. 2021Perturbation CheckLists for Evaluating NLG Evaluation Metricscited by 49
  23. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  24. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  25. 2020On the Limitations of Cross-lingual Encoders as Exposed by Reference-Free Machine Translation Evaluationcited by 36
  26. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  27. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  28. 2021FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generationcited by 19
  29. 2022USCORE: An Effective Approach to Fully Unsupervised Evaluation Metrics for Machine Translationcited by 15
  30. 2022Layer or Representation Space: What makes BERT-based Evaluation Metrics Robust?cited by 10

Built on by

Cited by year

Citing papers per year: 2024: 2; 2025: 2.

4 papers.

  1. 2025Interpretable Text Embeddings and Text Similarity Explanation: A Surveycited by 12
  2. 2024xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metricscited by 7
  3. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metricscited by 4
  4. 2025BatchGEMBA: Token-Efficient Machine Translation Evaluation with Batched Prompting and Prompt Compressioncited by 3