mlsanityData to 14 Oct 2026

Exploring the features used for summary evaluation by Human and GPT

arXiv 2512.19620 ↗First posted 22 Dec 2025cs.CL cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Summary assessment involves evaluating how well a generated summary reflects the key ideas and meaning of the source text, requiring a deep understanding of the content. Large Language Models (LLMs) have been used to automate this process, acting as judges to evaluate summaries with respect to the original text. While previous research investigated the alignment between LLMs and Human responses, it is not yet well understood what properties or features are exploited by them when asked to evaluate based on a particular quality dimension, and there has not been much attention towards mapping between evaluation scores and metrics. In this paper, we address this issue and discover features aligned with Human and Generative Pre-trained Transformers (GPTs) responses by studying statistical and machine learning metrics. Furthermore, we show that instructing GPTs to employ metrics used by Human can improve their judgment and conforming them better with human responses.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 22 Dec 2025.

Builds on

20 of 27 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  4. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  5. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  6. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  7. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  8. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 422
  9. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 418
  10. 2024An Empirical Study of LLM-as-a-Judge for LLM Evaluation: Fine-tuned Judge Model is not a General Substitute for GPT-4cited by 94
  11. 2018BLEU is Not Suitable for the Evaluation of Text Simplificationcited by 78
  12. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 37
8 more
  1. 2022CTRLEval: An Unsupervised Reference-Free Metric for Evaluating Controlled Text Generationcited by 29
  2. 2024Self-rationalization improves LLM as a fine-grained judgecited by 27
  3. 2023INSTRUCTSCORE: Explainable Text Generation Evaluation with Finegrained Feedbackcited by 22
  4. 2024Themis: A Reference-free NLG Evaluation Language Model with Flexibility and Interpretabilitycited by 19
  5. 2022SMART: Sentences as Basic Units for Text Evaluationcited by 18
  6. 2025Reassessing Extractive QA Datasets at Scale: LLM-as-a-Judge and In-Depth Analysescited by 14
  7. 2023Fusion-Eval: Integrating Assistant Evaluators with LLMscited by 6
  8. 2025J1: Exploring Simple Test-Time Scaling for LLM-as-a-Judgecited by 5

Built on by

Not cited yet.