mlsanityData to 7 Oct 2026

TriSum: Learning Summarization Ability from Large Language Models with Structured Rationale

arXiv 2403.10351 ↗First posted 15 Mar 2024cs.CLcited by 10

Not on the map. Why?

Abstract (arXiv)

The advent of large language models (LLMs) has significantly advanced natural language processing tasks like text summarization. However, their large size and computational demands, coupled with privacy concerns in data transmission, limit their use in resource-constrained and privacy-centric settings. To overcome this, we introduce TriSum, a framework for distilling LLMs' text summarization abilities into a compact, local model. Initially, LLMs extract a set of aspect-triple rationales and summaries, which are refined using a dual-scoring method for quality. Next, a smaller local model is trained with these tasks, employing a curriculum learning strategy that evolves from simple to complex tasks. Our method enhances local model performance on various benchmarks (CNN/DailyMail, XSum, and ClinicalTrial), outperforming baselines by 4.5%, 8.5%, and 7.4%, respectively. It also improves interpretability by providing insights into the summarization rationale.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 39 of its 59 references that are in the map. First posted on arXiv 15 Mar 2024.

Builds on

39 of 59 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  8. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  9. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  10. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  11. 2017Towards A Rigorous Science of Interpretable Machine Learningcited by 1,400
  12. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
27 more
  1. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  2. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  5. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,026
  6. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  7. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  8. 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 710
  9. 2019Text Summarization with Pretrained Encoderscited by 553
  10. 2022Large Language Models Are Reasoning Teacherscited by 318
  11. 2019On The Power of Curriculum Learning in Training Deep Networkscited by 285
  12. 2020ReClor: A Reading Comprehension Dataset Requiring Logical Reasoningcited by 211
  13. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  14. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  15. 2022Distilling Reasoning Capabilities into Smaller Language Modelscited by 155
  16. 2019End-to-End Open-Domain Question Answering with BERTserinicited by 152
  17. 2019Distilling Task-Specific Knowledge from BERT into Simple Neural Networkscited by 152
  18. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 131
  19. 2020GSum: A General Framework for Guided Neural Abstractive Summarizationcited by 129
  20. 2023Exploring the Limits of ChatGPT for Query or Aspect-based Text Summarizationcited by 122
  21. 2023TrueTeacher: Learning Factual Consistency Evaluation with Large Language Modelscited by 82
  22. 2023On Learning to Summarize with Large Language Models as Referencescited by 80
  23. 2020PLATO-2: Towards Building an Open-Domain Chatbot via Curriculum Learningcited by 79
  24. 2020Pre-trained Summarization Distillationcited by 70
  25. 2022PINTO: Faithful Language Reasoning Using Prompt-Generated Rationalescited by 56
  26. 2021Sequence Level Contrastive Learning for Text Summarizationcited by 31
  27. 2021Attention Temperature Matters in Abstractive Summarization Distillationcited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 2; 2026: 2.

10 papers.

  1. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  2. 2024Panacea: A foundation model for clinical trial search, summarization, design, and recruitmentcited by 10
  3. 2025Trustworthy Summarization via Uncertainty Quantification and Risk Awareness in Large Language Modelscited by 10
  4. 2024ReasoningRank: Teaching Student Models to Rank through Reasoning-Based Knowledge Distillationcited by 9
  5. 2024Just What You Desire: Constrained Timeline Summarization with Self-Reflection for Enhanced Relevancecited by 7
  6. 2024Factual Dialogue Summarization via Learning from Large Language Modelscited by 6
  7. 2024How Private are Language Models in Abstractive Summarization?cited by 2
  8. 2025Natural Language Processing in Support of Evidence-based Medicine: A Scoping Reviewcited by 2
  9. 2026SCURank: Ranking Multiple Candidate Summaries with Summary Content Units for Enhanced Summarizationcited by 0
  10. 2026Improving Argument Saliency Coverage in Small LLMs for Long Legal Opinion Summarization via Sequence-Level Distillationcited by 0