mlsanityData to 7 Oct 2026

Multi-Dimensional Evaluation of Text Summarization with In-Context Learning

arXiv 2306.01200 ↗First posted 1 Jun 2023cs.CLcited by 44

Not on the map. Why?

Abstract (arXiv)

Evaluation of natural language generation (NLG) is complex and multi-dimensional. Generated text can be evaluated for fluency, coherence, factuality, or any other dimensions of interest. Most frameworks that perform such multi-dimensional evaluation require training on large manually or synthetically generated datasets. In this paper, we study the efficacy of large language models as multi-dimensional evaluators using in-context learning, obviating the need for large training datasets. Our experiments show that in-context learning-based evaluators are competitive with learned evaluation frameworks for the task of text summarization, establishing state-of-the-art on dimensions such as relevance and factual consistency. We then analyze the effects of factors such as the selection and number of in-context examples on performance. Finally, we study the efficacy of in-context learning based evaluators in evaluating zero-shot summaries written by large language models such as GPT-3.

Key facts. Cited by 44 papers in the map (data through 7 Oct 2026). Builds on 17 of its 20 references that are in the map. First posted on arXiv 1 Jun 2023.

Builds on

17 of 20 references.

References · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  3. 2022Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?cited by 1,413
  4. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  5. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  6. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  7. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  9. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 415
  10. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 324
  12. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
5 more
  1. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  2. 2020GRADE: Automatic Graph-Enhanced Coherence Metric for Evaluating Open-Domain Dialogue Systemscited by 76
  3. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  4. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  5. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 8; 2024: 20; 2025: 13; 2026: 3.

44 papers.

Sort:

By area:

  1. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  2. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 480
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  5. 2023The Devil is in the Errors: Leveraging Large Language Models for Fine-grained Machine Translation Evaluationcited by 95
  6. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 71
  7. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  8. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  9. 2024Evaluation Agent: Efficient and Promptable Evaluation Framework for Visual Generative Modelscited by 41
  10. 2024Membership Inference Attacks Against In-Context Learningcited by 38
  11. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  12. 2024Calibrating Long-form Generations from Large Language Modelscited by 31
32 more
  1. 2023On Context Utilization in Summarization with Large Language Modelscited by 30
  2. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  3. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  4. 2025The Scales of Justitia: A Comprehensive Survey on Safety Evaluation of LLMscited by 24
  5. 2023Exploring the Reliability of Large Language Models as Customized Evaluators for Diverse NLP Taskscited by 23
  6. 2024FFN-SkipLLM: A Hidden Gem for Autoregressive Decoding with Adaptive Feed Forward Skippingcited by 23
  7. 2024Improving Context-Aware Preference Modeling for Language Modelscited by 22
  8. 2024What Makes a Good Story and How Can We Measure It? A Comprehensive Survey of Story Evaluationcited by 17
  9. 2024Towards a Unified View of Preference Learning for Large Language Models: A Surveycited by 17
  10. 2023Walking a Tightrope -- Evaluating Large Language Models in High-Risk Domainscited by 15
  11. 2024Assessing and Verifying Task Utility in LLM-Powered Applicationscited by 15
  12. 2025MAPLE: Many-Shot Adaptive Pseudo-Labeling for In-Context Learningcited by 13
  13. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  14. 2025SelfReflect: Can LLMs Communicate Their Internal Answer Distribution?cited by 6
  15. 2025An Evaluation of Large Language Models on Text Summarization Tasks Using Prompt Engineering Techniquescited by 6
  16. 2023Linking Surface Facts to Large-Scale Knowledge Graphscited by 5
  17. 2024Towards better Human-Agent Alignment: Assessing Task Utility in LLM-Powered Applicationscited by 5
  18. 2025HPSS: Heuristic Prompting Strategy Search for LLM Evaluatorscited by 5
  19. 2024Redefining Proactivity for Information Seeking Dialoguecited by 4
  20. 2023Grounded Intuition of GPT-Vision's Abilities with Scientific Imagescited by 3
  21. 2024PerSEval: Assessing Personalization in Text Summarizerscited by 3
  22. 2026K-Sort Eval: Efficient Preference Evaluation for Visual Generation via Corrected VLM-as-a-Judgecited by 3
  23. 2024Aligning Large Language Models from Self-Reference AI Feedback with one General Principlecited by 2
  24. 2025Consistency Evaluation of News Article Summaries Generated by Large (and Small) Language Modelscited by 2
  25. 2025Group Preference Alignment: Customized LLM Response Generation from In-Situ Conversationscited by 2
  26. 2025Aspect-Based Summarization with Self-Aspect Retrieval Enhanced Generationcited by 2
  27. 2025Reranking-based Generation for Unbiased Perspective Summarizationcited by 2
  28. 2026Open Evaluation Agent: Efficient and Promptable Evaluation of Visual Generative Modelscited by 2
  29. 2025From Cross-Task Examples to In-Task Prompts: A Graph-Based Pseudo-Labeling Framework for In-context Learningcited by 1
  30. 2024A Dataset for Evaluating LLM-based Evaluation Functions for Research Question Extraction Taskcited by 0
  31. 2025TextMineX: Data, Evaluation Framework and Ontology-guided LLM Pipeline for Humanitarian Mine Actioncited by 0
  32. 2026Probabilistic Guarantees for Reducing Contextual Hallucinations in LLMscited by 0