mlsanityData to 7 Oct 2026

The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metrics

arXiv 2310.19792 ↗First posted 30 Oct 2023cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

With an increasing number of parameters and pre-training data, generative large language models (LLMs) have shown remarkable capabilities to solve tasks with minimal or no task-related examples. Notably, LLMs have been successfully employed as evaluation metrics in text generation tasks. Within this context, we introduce the Eval4NLP 2023 shared task that asks participants to explore prompting and score extraction for machine translation (MT) and summarization evaluation. Specifically, we propose a novel competition setting in which we select a list of allowed LLMs and disallow fine-tuning to ensure a focus on prompting. We present an overview of participants' approaches and evaluate them on a new reference-free test set spanning three language pairs for MT and a summarization dataset. Notably, despite the task's restrictions, the best-performing systems achieve results on par with or even surpassing recent reference-free metrics developed using larger models, including GEMBA and Comet-Kiwi-XXL. Finally, as a separate track, we perform a small-scale human evaluation of the plausibility of explanations given by the LLMs.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 53 of its 95 references that are in the map. First posted on arXiv 30 Oct 2023.

Builds on

53 of 95 references.

References · 53

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  4. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  11. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  12. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
38 more
  1. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  2. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  3. 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
  4. 2020COMET: A Neural Framework for MT Evaluationcited by 1,024
  5. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  8. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  9. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  10. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  11. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  12. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  13. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  14. 2021Experts, Errors, and Context: A Large-Scale Study of Human Evaluation for Machine Translationcited by 333
  15. 2020Evaluation of Text Generation: A Surveycited by 275
  16. 2023Mathematical Capabilities of ChatGPTcited by 263
  17. 2023xCOMET: Transparent Machine Translation Evaluation through Fine-grained Error Detectioncited by 260
  18. 2023Evaluating the Logical Reasoning Ability of ChatGPT and GPT-4cited by 202
  19. 2020A Survey of Evaluation Metrics Used for NLG Systemscited by 164
  20. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 141
  21. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  22. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  23. 2023Scaling up COMETKIWI: Unbabel-IST 2023 Submission for the Quality Estimation Shared Taskcited by 88
  24. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Modelscited by 87
  25. 2018A Study of Reinforcement Learning for Neural Machine Translationcited by 74
  26. 2023AutomaTikZ: Text-Guided Synthesis of Scientific Vector Graphics with TikZcited by 74
  27. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 71
  28. 2020Pre-trained Summarization Distillationcited by 70
  29. 2020SUPERT: Towards New Frontiers in Unsupervised Evaluation Metrics for Multi-Document Summarizationcited by 69
  30. 2021Codabench: Flexible, Easy-to-Use and Reproducible Benchmarking Platformcited by 65
  31. 2023ChatGPT: A Meta-Analysis after 2.5 Monthscited by 50
  32. 2022On the Blind Spots of Model-Based Evaluation Metrics for Text Generationcited by 48
  33. 2021The Eval4NLP Shared Task on Explainable Quality Estimation: Overview and Resultscited by 27
  34. 2021Global Explainability of BERT-Based Evaluation Metrics by Disentangling along Linguistic Factorscited by 25
  35. 2023Exploring ChatGPT's Ability to Rank Content: A Preliminary Study on Consistency with Human Preferencescited by 25
  36. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 21
  37. 2021FrugalScore: Learning Cheaper, Lighter and Faster Evaluation Metricsfor Automatic Text Generationcited by 19
  38. 2023The Inside Story: Towards Better Understanding of Machine Translation Neural Evaluation Metricscited by 15

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 14; 2025: 4; 2026: 1.

23 papers.

Sort:

By area:

  1. 2024MetricX-24: The Google Submission to the WMT 2024 Metrics Shared Taskcited by 92
  2. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  3. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  4. 2024How Good Are LLMs for Literary Translation, Really? Literary Translation Evaluation with Humans and LLMscited by 23
  5. 2024MQM-APE: Toward High-Quality Error Annotation Predictors with Automatic Post-Editing in LLM Translation Evaluatorscited by 22
  6. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  7. 2024Evaluating Diversity in Automatic Poetry Generationcited by 17
  8. 2024PrExMe! Large Scale Prompt Exploration of Open Source LLMs for Machine Translation and Summarization Evaluationcited by 15
  9. 2023Which is better? Exploring Prompting Strategy For LLM-based Metricscited by 13
  10. 2024ScImage: How Good Are Multimodal Large Language Models at Scientific Text-to-Image Generation?cited by 13
  11. 2025MetricX-25 and GemSpanEval: Google Translate Submissions to the WMT25 Evaluation Shared Taskcited by 10
  12. 2023Evaluating Task-oriented Dialogue Systems: A Systematic Review of Measures, Constructs and their Operationalisationscited by 9
11 more
  1. 2024Natural Language Processing RELIES on Linguisticscited by 9
  2. 2023Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Taskcited by 8
  3. 2024xCOMET-lite: Bridging the Gap Between Efficiency and Quality in Learned MT Evaluation Metricscited by 7
  4. 2025LLMs as Span Annotators: A Comparative Study of LLMs and Humanscited by 7
  5. 2023Exploring Prompting Large Language Models as Explainable Metricscited by 4
  6. 2024Evaluating Large Language Models for Structured Science Summarization in the Open Research Knowledge Graphcited by 4
  7. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metricscited by 4
  8. 2025Argument Summarization and its Evaluation in the Era of Large Language Modelscited by 4
  9. 2025Evaluating the Evaluator: Summarization Metrics and LLM-Judges beyond Englishcited by 2
  10. 2024What Would You Ask When You First Saw a2+b2=c2? Evaluating LLM on Curiosity-Driven Questioningcited by 1
  11. 2026ExCAM: Explainable Cultural Awareness Metricscited by 0