mlsanityData to 7 Oct 2026

Little Giants: Exploring the Potential of Small LLMs as Evaluation Metrics in Summarization in the Eval4NLP 2023 Shared Task

arXiv 2311.00686 ↗First posted 1 Nov 2023cs.CLcited by 8

Not on the map. Why?

Abstract (arXiv)

This paper describes and analyzes our participation in the 2023 Eval4NLP shared task, which focuses on assessing the effectiveness of prompt-based techniques to empower Large Language Models to handle the task of quality estimation, particularly in the context of evaluating machine translations and summaries. We conducted systematic experiments with various prompting techniques, including standard prompting, prompts informed by annotator instructions, and innovative chain-of-thought prompting. In addition, we integrated these approaches with zero-shot and one-shot learning methods to maximize the efficacy of our evaluation procedures. Our work reveals that combining these approaches using a "small", open source model (orca_mini_v3_7B) yields competitive results.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 12 of its 18 references that are in the map. First posted on arXiv 1 Nov 2023.

Builds on

12 of 18 references.

References · 12

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  3. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  4. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  5. 2019Defending Against Neural Fake Newscited by 553
  6. 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
  7. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  8. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  9. 2020Explainable Automated Fact-Checking: A Surveycited by 83
  10. 2023The Eval4NLP 2023 Shared Task on Prompting Large Language Models as Explainable Metricscited by 23
  11. 2023Simple LLM Prompting is State-of-the-Art for Robust and Multilingual Dialogue Evaluationcited by 19
  12. 2023Knowledge-Prompted Estimator: A Novel Approach to Explainable Machine Translation Assessmentcited by 3

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 4.

8 papers.

  1. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  4. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  5. 2024HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agentcited by 24
  6. 2025HPSS: Heuristic Prompting Strategy Search for LLM Evaluatorscited by 5
  7. 2025Scoring, Reasoning, and Selecting the Best! Ensembling Large Language Models via a Peer-Review Processcited by 4
  8. 2025Learning to Align Multi-Faceted Evaluation: A Unified and Robust Frameworkcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024PromptOptMe: Error-Aware Prompt Compression for LLM-based MT Evaluation Metrics
  2. 2023Large Language Models are Not Yet Human-Level Evaluators for Abstractive Summarization
  3. 2024What do Large Language Models Need for Machine Translation Evaluation?
  4. 2023Error Analysis Prompting Enables Human-Like Translation Evaluation in Large Language Models
  5. 2023Evaluating Factual Consistency of Summaries with Large Language Models
  6. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Tasks
  7. 2024Source Code Summarization in the Era of Large Language Models
  8. 2024Thinking Fair and Slow: On the Efficacy of Structured Prompts for Debiasing Language Models