mlsanityData to 13 Oct 2026

Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method

arXiv 2305.13412 ↗First posted 22 May 2023cs.CLcited by 76

On the map: LLM reasoning → Reasoning benchmarks and evaluation

Abstract (arXiv)

Automatic summarization generates concise summaries that contain key ideas of source documents. As the most mainstream datasets for the news sub-domain, CNN/DailyMail and BBC XSum have been widely used for performance benchmarking. However, the reference summaries of those datasets turn out to be noisy, mainly in terms of factual hallucination and information redundancy. To address this challenge, we first annotate new expert-writing Element-aware test sets following the "Lasswell Communication Model" proposed by Lasswell (1948), allowing reference summaries to focus on more fine-grained news elements objectively and comprehensively. Utilizing the new test sets, we observe the surprising zero-shot summary ability of LLMs, which addresses the issue of the inconsistent results between human preference and automatic evaluation metrics of LLMs' zero-shot summaries in prior work. Further, we propose a Summary Chain-of-Thought (SumCoT) technique to elicit LLMs to generate summaries step by step, which helps them integrate more fine-grained details of source documents into the final summaries that correlate with the human writing mindset. Experimental results show our method outperforms state-of-the-art fine-tuned PLMs and zero-shot LLMs by +4.33/+4.77 in ROUGE-L on the two datasets, respectively. Dataset and code are publicly available at https://github.com/Alsace08/SumCoT.

Key facts. Cited by 76 papers in the map (data through 13 Oct 2026). Placed in Reasoning benchmarks and evaluation, one of 17 directions in LLM reasoning. Builds on 47 of its 67 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

47 of 67 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Show Your Work: Scratchpads for Intermediate Computation with Language Models30 Nov

Other references · 46

  1. 2017Attention Is All You Needcited by 60,106
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  3. 2020Language Models are Few-Shot Learnerscited by 32,320
  4. 2022Training language models to follow instructions with human feedbackcited by 15,856
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  7. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  9. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,657
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,317
  12. 2022Large Language Models are Zero-Shot Reasonerscited by 4,348
34 more
  1. 2014Sequence to Sequence Learning with Neural Networkscited by 2,981
  2. 2020Learning to summarize from human feedbackcited by 2,575
  3. 2022Emergent Abilities of Large Language Modelscited by 2,499
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,468
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,404
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  7. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,200
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  9. 2015Teaching Machines to Read and Comprehendcited by 1,045
  10. 2020COMET: A Neural Framework for MT Evaluationcited by 1,027
  11. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 898
  12. 2015Pointer Networkscited by 884
  13. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  14. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 746
  15. 2022Automatic Chain of Thought Prompting in Large Language Modelscited by 687
  16. 2020SummEval: Re-evaluating Summarization Evaluationcited by 668
  17. 2022Language Models are Multilingual Chain-of-Thought Reasonerscited by 525
  18. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  19. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 429
  20. 2019MoverScore: Text Generation Evaluating with Contextualized Embeddings and Earth Mover Distancecited by 416
  21. 2022News Summarization and Evaluation in the Era of GPT-3cited by 326
  22. 2018Newsroom: A Dataset of 1.3 Million Summaries with Diverse Extractive Strategiescited by 293
  23. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 204
  24. 2019Neural Text Summarization: A Critical Evaluationcited by 189
  25. 2018Bottom-Up Abstractive Summarizationcited by 177
  26. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  27. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  28. 2019Fine-tune BERT for Extractive Summarizationcited by 122
  29. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  30. 2020CTRLsum: Towards Generic Controllable Text Summarizationcited by 98
  31. 2021Planning with Learned Entity Prompts for Abstractive Summarizationcited by 75
  32. 2015Better Summarization Evaluation with Word Embeddings for ROUGEcited by 66
  33. 2022A Well-Composed Text is Half Done! Composition Sampling for Diverse Conditional Generationcited by 28
  34. 2017Reader-Aware Multi-Document Summarization: An Enhanced Model and The First Datasetcited by 6

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2023: 12; 2024: 35; 2025: 19; 2026: 10.

76 papers.

Sort:

By area:

  1. 2023PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimizationcited by 313
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 280
  3. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 243
  4. 2023Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Futurecited by 184
  5. 2025LlamaV-o1: Rethinking Step-by-step Visual Reasoning in LLMscited by 145
  6. 2024Large Language Models Meet NLP: A Surveycited by 103
  7. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  8. 2023Igniting Language Intelligence: The Hitchhiker's Guide From Chain-of-Thought Reasoning to Language Agentscited by 72
  9. 2024Infinite-LLM: Efficient LLM Service for Long Context with DistAttention and Distributed KVCachecited by 71
  10. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  11. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  12. 2023Large Language Models are Temporal and Causal Reasoners for Video Question Answeringcited by 51
38 more
  1. 2023Summarization is (Almost) Deadcited by 44
  2. 2024Source Code Summarization in the Era of Large Language Modelscited by 31
  3. 2023On Context Utilization in Summarization with Large Language Modelscited by 30
  4. 2024FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extractioncited by 28
  5. 2024Beemo: Benchmark of Expert-edited Machine-generated Outputscited by 26
  6. 2024Are Large Language Models Reliable Argument Quality Annotators?cited by 24
  7. 2023PaniniQA: Enhancing Patient Education Through Interactive Question Answeringcited by 21
  8. 2024Enhancing Interactive Image Retrieval With Query Rewriting Using Large Language Models and Vision Language Modelscited by 19
  9. 2024InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inferencecited by 19
  10. 2024Embedding Trajectory for Out-of-Distribution Detection in Mathematical Reasoningcited by 18
  11. 2024Scaling Up Video Summarization Pretraining with Large Language Modelscited by 17
  12. 2024ToxiCraft: A Novel Framework for Synthetic Generation of Harmful Informationcited by 17
  13. 2023Struc-Bench: Are Large Language Models Really Good at Generating Complex Structured Data?cited by 16
  14. 2024Are AI-Generated Text Detectors Robust to Adversarial Perturbations?cited by 16
  15. 2024Position: Towards Bidirectional Human-AI Alignmentcited by 15
  16. 2025Fate: Fast Edge Inference of Mixture-of-Experts Models via Cross-Layer Gatecited by 13
  17. 2024ULTRA: Unleash LLMs' Potential for Event Argument Extraction through Hierarchical Modeling and Pair-wise Self-Refinementcited by 12
  18. 2023Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Modelscited by 11
  19. 2024Are Large Language Models Good Classifiers? A Study on Edit Intent Classification in Scientific Document Revisionscited by 11
  20. 2023NLEBench+NorGLM: A Comprehensive Empirical Analysis and Benchmark Dataset for Generative Language Models in Norwegiancited by 9
  21. 2024Key-Element-Informed sLLM Tuning for Document Summarizationcited by 8
  22. 2024Truth or Mirage? Towards End-to-End Factuality Evaluation with LLM-Oasiscited by 8
  23. 2024Combining Hierachical VAEs with LLMs for clinically meaningful timeline summarisation in social mediacited by 7
  24. 2024Improving Clinical Note Generation from Complex Doctor-Patient Conversationcited by 7
  25. 2025Fast-Slow-Thinking: Complex Task Solving with Large Language Modelscited by 7
  26. 2023Improving Factual Consistency of News Summarization by Contrastive Preference Optimizationcited by 6
  27. 2024Can Large Language Model Summarizers Adapt to Diverse Scientific Communication Goals?cited by 6
  28. 2024Multi-Dimensional Optimization for Text Summarization via Reinforcement Learningcited by 6
  29. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
  30. 2025Exploring Large Language Models for Semantic Analysis and Categorization of Android Malwarecited by 6
  31. 2024SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrievalcited by 5
  32. 2024AlignSum: Data Pyramid Hierarchical Fine-tuning for Aligning with Human Summarization Preferencecited by 4
  33. 2024DomainSum: A Hierarchical Benchmark for Fine-Grained Domain Shift in Abstractive Text Summarizationcited by 4
  34. 2025QA-prompting: Improving Summarization with Large Language Models using Question-Answeringcited by 4
  35. 2025Explainable Mapper: Charting LLM Embedding Spaces Using Perturbation-Based Explanation and Verification Agentscited by 4
  36. 2024Building Accurate Translation-Tailored LLMs with Language Aware Instruction Tuningcited by 3
  37. 2024Prompting for Numerical Sequences: A Case Study on Market Comment Generationcited by 3
  38. 2024EVA-Score: Evaluating Abstractive Long-form Summarization on Informativeness through Extraction and Validationcited by 3

50 most cited of 76 All 76 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Information-Theoretic Distillation for Reference-less Summarization
  2. 2024Flexible and Adaptable Summarization via Expertise Separation
  3. 2025Towards Multi-dimensional Evaluation of LLM Summarization across Domains and Languages
  4. 2025Multi-round, Chain-of-thought Post-editing for Unfaithful Summaries
  5. 2025Scaling Multi-Document Event Summarization: Evaluating Compression vs. Full-Text Approaches
  6. 2026A Stepwise Questioning Expert-Editor Multi-Agent Framework for Long-Document Summarization
  7. 2024Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark
  8. 2025StrucSum: Graph-Structured Reasoning for Long Document Extractive Summarization with LLMs