mlsanityData to 14 Oct 2026

Document Summarization with Conformal Importance Guarantees

arXiv 2509.20461 ↗First posted 24 Sep 2025cs.CL cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

Automatic summarization systems have advanced rapidly with large language models (LLMs), yet they still lack reliable guarantees on inclusion of critical content in high-stakes domains like healthcare, law, and finance. In this work, we introduce Conformal Importance Summarization, the first framework for importance-preserving summary generation which uses conformal prediction to provide rigorous, distribution-free coverage guarantees. By calibrating thresholds on sentence-level importance scores, we enable extractive document summarization with user-specified coverage and recall rates over critical content. Our method is model-agnostic, requires only a small calibration set, and seamlessly integrates with existing black-box LLMs. Experiments on established summarization benchmarks demonstrate that Conformal Importance Summarization achieves the theoretically assured information coverage rate. Our work suggests that Conformal Importance Summarization can be combined with existing techniques to achieve reliable, controllable automatic summarization, paving the way for safer deployment of AI summarization tools in critical applications. Code is available at https://github.com/layer6ai-labs/conformal-importance-summarization.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 46 of its 69 references that are in the map. First posted on arXiv 24 Sep 2025.

Builds on

46 of 69 references.

References · 46

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2024The Llama 3 Herd of Modelscited by 14,715
  6. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  11. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  12. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
34 more
  1. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  2. 2015Teaching Machines to Read and Comprehendcited by 1,049
  3. 2021A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantificationcited by 792
  4. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  5. 2019Conformalized Quantile Regressioncited by 629
  6. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  7. 2019Text Summarization with Pretrained Encoderscited by 556
  8. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  9. 2020Classification with Valid and Adaptive Coveragecited by 396
  10. 2020Uncertainty Sets for Image Classifiers using Conformal Predictioncited by 366
  11. 2017A Deep Reinforced Model for Abstractive Summarizationcited by 365
  12. 2022Conformal Risk Controlcited by 299
  13. 2018Bottom-Up Abstractive Summarizationcited by 177
  14. 2023Conformal Language Modelingcited by 170
  15. 2018Fast Abstractive Summarization with Reinforce-Selected Sentence Rewritingcited by 158
  16. 2023Conformal Prediction with Large Language Models for Multi-Choice Question Answeringcited by 139
  17. 2023BooookScore: A systematic exploration of book-length summarization in the era of LLMscited by 137
  18. 2020TLDR: Extreme Summarization of Scientific Documentscited by 136
  19. 2024Language Models with Conformal Factuality Guaranteescited by 131
  20. 2024Large language model validity via enhanced conformal prediction methodscited by 92
  21. 2019Nested conformal prediction and quantile out-of-bag ensemble methodscited by 89
  22. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  23. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  24. 2021Copula-based conformal prediction for Multi-Target Regressioncited by 53
  25. 2023Conformal Prediction for Deep Classifier via Label Rankingcited by 53
  26. 2022On the Limitations of Reference-Free Evaluations of Generated Textcited by 52
  27. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  28. 2024Conformal Prediction Sets Improve Human Decision Makingcited by 35
  29. 2021CSDS: A Fine-Grained Chinese Dataset for Customer Service Dialogue Summarizationcited by 27
  30. 2024Conformal Prediction Sets Can Cause Disparate Impactcited by 14
  31. 2025Conformal Language Model Reasoning with Coherent Factualitycited by 14
  32. 2014Efficiency of conformalized ridge regressioncited by 13
  33. 2025Development and Validation of the Provider Documentation Summarization Quality Instrument for Large Language Modelscited by 9
  34. 2025Response Quality Assessment for Retrieval-Augmented Generation via Conditional Conformal Factualitycited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 6.

6 papers.

  1. 2026Conformal Agent Error Attributioncited by 4
  2. 2026CARE: A Conformal Safety Layer for Medical Summarizationcited by 2
  3. 2026When Summaries Distort Decisions: Information Fidelity in LLM-Compressed Financial Analysiscited by 2
  4. 2026RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generatorcited by 1
  5. 2026Conf-Gen: Conformal Uncertainty Quantification for Generative Modelscited by 1
  6. 2026Unifying Conformal Language Tasks with In-Context Ensemblescited by 0