# What's Wrong? Refining Meeting Summaries with LLM Feedback

arXiv: https://arxiv.org/abs/2407.11919 · HTML: https://mlsanity.com/paper/2407.11919/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 16 Jul 2024.

## Builds on (most cited first, 30 of 31)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone](https://mlsanity.com/paper/2404.14219/) (2024)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Large Language Models Cannot Self-Correct Reasoning Yet](https://mlsanity.com/paper/2310.01798/) (2023)
- [Self-critiquing models for assisting human evaluators](https://mlsanity.com/paper/2206.05802/) (2022)
- [Language Models can Solve Computer Tasks](https://mlsanity.com/paper/2303.17491/) (2023)
- [QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarization](https://mlsanity.com/paper/2104.05938/) (2021)
- [Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategies](https://mlsanity.com/paper/2308.03188/) (2023)
- [BooookScore: A systematic exploration of book-length summarization in the era of LLMs](https://mlsanity.com/paper/2310.00785/) (2023)
- [Efficient Long-Text Understanding with Short-Text Models](https://mlsanity.com/paper/2208.00748/) (2022)
- [SummIt: Iterative Text Summarization via ChatGPT](https://mlsanity.com/paper/2305.14835/) (2023)
- [Investigating Efficiently Extending Transformers for Long Input Summarization](https://mlsanity.com/paper/2208.04347/) (2022)
- [TEaR: Improving LLM-based Machine Translation with Systematic Self-Refinement](https://mlsanity.com/paper/2402.16379/) (2024)
- [Building Real-World Meeting Summarization Systems using Large Language Models: A Practical Perspective](https://mlsanity.com/paper/2310.19233/) (2023)
- [Think about it! Improving defeasible reasoning by first modeling the question scenario](https://mlsanity.com/paper/2110.12349/) (2021)
- [We are Who We Cite: Bridges of Influence Between Natural Language Processing and Other Academic Fields](https://mlsanity.com/paper/2310.14870/) (2023)
- [LLM-ARC: Enhancing LLMs with an Automated Reasoning Critic](https://mlsanity.com/paper/2406.17663/) (2024)
- [A Bag of Tricks for Dialogue Summarization](https://mlsanity.com/paper/2109.08232/) (2021)
- [Paraphrase Detection: Human vs. Machine Content](https://mlsanity.com/paper/2303.13989/) (2023)
- [What's under the hood: Investigating Automatic Metrics on Meeting Summarization](https://mlsanity.com/paper/2404.11124/) (2024)
- [CADS: A Systematic Literature Review on the Challenges of Abstractive Dialogue Summarization](https://mlsanity.com/paper/2406.07494/) (2024)
- [N-Critics: Self-Refinement of Large Language Models with Ensemble of Critics](https://mlsanity.com/paper/2310.18679/) (2023)
- [An End-to-End Dialogue Summarization System for Sales Calls](https://mlsanity.com/paper/2204.12951/) (2022)
- [Learning to Correct for QA Reasoning with Black-box LLMs](https://mlsanity.com/paper/2406.18695/) (2024)
- [A Context-Enhanced De-identification System](https://mlsanity.com/paper/2102.08513/) (2021)

## Built on by (most cited first, 8 of 8)

- [Stay Focused: Problem Drift in Multi-Agent Debate](https://mlsanity.com/paper/2502.19559/) (2025)
- [Entropy-Gated Branching for Efficient Test-Time Reasoning](https://mlsanity.com/paper/2503.21961/) (2025)
- [Tell me what I need to know: Exploring LLM-based (Personalized) Abstractive Multi-Source Meeting Summarization](https://mlsanity.com/paper/2410.14545/) (2024)
- [Loki's Dance of Illusions: A Comprehensive Survey of Hallucination in Large Language Models](https://mlsanity.com/paper/2507.02870/) (2025)
- [Is my Meeting Summary Good? Estimating Quality with a Multi-LLM Evaluator](https://mlsanity.com/paper/2411.18444/) (2024)
- [You need to MIMIC to get FAME: Solving Meeting Transcript Scarcity with a Multi-Agent Conversations](https://mlsanity.com/paper/2502.13001/) (2025)
- [FactsR: A Safer Method for Producing High Quality Healthcare Documentation](https://mlsanity.com/paper/2505.10360/) (2025)
- [Token Distribution versus Data Volume: Domain Balancing in Multi-Domain Meeting Summarisation](https://mlsanity.com/paper/2608.15935/) (2026)

## Abstract (arXiv)

Meeting summarization has become a critical task since digital encounters have become a common practice. Large language models (LLMs) show great potential in summarization, offering enhanced coherence and context understanding compared to traditional methods. However, they still struggle to maintain relevance and avoid hallucination. We introduce a multi-LLM correction approach for meeting summarization using a two-phase process that mimics the human review process: mistake identification and summary refinement. We release QMSum Mistake, a dataset of 200 automatically generated meeting summaries annotated by humans on nine error types, including structural, omission, and irrelevance errors. Our experiments show that these errors can be identified with high accuracy by an LLM. We transform identified mistakes into actionable feedback to improve the quality of a given summary measured by relevance, informativeness, conciseness, and coherence. This post-hoc refinement effectively improves summary quality by leveraging multiple LLMs to validate output quality. Our multi-LLM approach for meeting summarization shows potential for similar complex text generation tasks requiring robustness, action planning, and discussion towards a goal.