mlsanityData to 14 Oct 2026

Towards a Robust Retrieval-Based Summarization System

arXiv 2403.19889 ↗First posted 29 Mar 2024cs.CL cs.AI cs.IRcited by 5

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

This paper describes an investigation of the robustness of large language models (LLMs) for retrieval augmented generation (RAG)-based summarization tasks. While LLMs provide summarization capabilities, their performance in complex, real-world scenarios remains under-explored. Our first contribution is LogicSumm, an innovative evaluation framework incorporating realistic scenarios to assess LLM robustness during RAG-based summarization. Based on limitations identified by LogiSumm, we then developed SummRAG, a comprehensive system to create training dialogues and fine-tune a model to enhance robustness within LogicSumm's scenarios. SummRAG is an example of our goal of defining structured methods to test the capabilities of an LLM, rather than addressing issues in a one-off fashion. Experimental results confirm the power of SummRAG, showcasing improved logical coherence and summarization quality. Data, corresponding model weights, and Python code are available online.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 31 of its 39 references that are in the map. First posted on arXiv 29 Mar 2024.

Builds on

31 of 39 references.

References · 31

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  8. 2020Scaling Laws for Neural Language Modelscited by 6,586
  9. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  11. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  12. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
19 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  3. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  4. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
  5. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  6. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  9. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 952
  10. 2019Text Summarization with Pretrained Encoderscited by 556
  11. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  12. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  13. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  14. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  15. 2021Retrieval Augmented Code Generation and Summarizationcited by 149
  16. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  17. 2024Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Modelscited by 93
  18. 2018Skeleton-to-Response: Dialogue Generation Guided by Retrieval Memorycited by 49
  19. 2024List-aware Reranking-Truncation Joint Model for Search and Retrieval-augmented Generationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Towards Trustworthy Retrieval Augmented Generation for Large Language Models: A Surveycited by 30
  2. 2024CROPS: A Deployable Crop Management System Over All Possible State Availabilitiescited by 0
  3. 2025A Unified Retrieval Framework with Document Ranking and EDU Filtering for Multi-document Summarizationcited by 0
  4. 2025Mitigating Hallucinations in Zero-Shot Scientific Summarisation: A Pilot Studycited by 0
  5. 2026Don't Scroll Back: Missing-Evidence Memory for Streaming Dialogue Summarizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating the Retrieval Robustness of Large Language Models
  2. 2024Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs
  3. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  4. 2025Estimating Optimal Context Length for Hybrid Retrieval-augmented Multi-document Summarization
  5. 2025Diversity Enhances an LLM's Performance in RAG and Long-context Task
  6. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation
  7. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  8. 2024Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG