# Abstractive Text Summarization Using the BRIO Training Paradigm

arXiv: https://arxiv.org/abs/2305.13696 · HTML: https://mlsanity.com/paper/2305.13696/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 31 references that are in the map.
- First posted on arXiv 23 May 2023.

## Builds on (most cited first, 17 of 17)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [mT5: A massively multilingual pre-trained text-to-text transformer](https://mlsanity.com/paper/2010.11934/) (2020)
- [Efficient Attentions for Long Document Summarization](https://mlsanity.com/paper/2104.02112/) (2021)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [BRIO: Bringing Order to Abstractive Summarization](https://mlsanity.com/paper/2203.16804/) (2022)
- [ViT5: Pretrained Text-to-Text Transformer for Vietnamese Language Generation](https://mlsanity.com/paper/2205.06457/) (2022)
- [BARTpho: Pre-trained Sequence-to-Sequence Models for Vietnamese](https://mlsanity.com/paper/2109.09701/) (2021)
- [Hierarchical Learning for Generation with Long Source Sequences](https://mlsanity.com/paper/2104.07545/) (2021)
- [Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2](https://mlsanity.com/paper/2006.01997/) (2020)
- [Deep Recurrent Generative Decoder for Abstractive Text Summarization](https://mlsanity.com/paper/1708.00625/) (2017)
- [Leveraging ParsBERT and Pretrained mT5 for Persian Abstractive Text Summarization](https://mlsanity.com/paper/2012.11204/) (2020)
- [Leveraging Locality in Abstractive Text Summarization](https://mlsanity.com/paper/2205.12476/) (2022)
- [Actor-Critic based Training Framework for Abstractive Summarization](https://mlsanity.com/paper/1803.11070/) (2018)

## Built on by (most cited first, 1 of 1)

- [RoLargeSum: A Large Dialect-Aware Romanian News Dataset for Summary, Headline, and Keyword Generation](https://mlsanity.com/paper/2412.11317/) (2024)

## Abstract (arXiv)

Summary sentences produced by abstractive summarization models may be coherent and comprehensive, but they lack control and rely heavily on reference summaries. The BRIO training paradigm assumes a non-deterministic distribution to reduce the model's dependence on reference summaries, and improve model performance during inference. This paper presents a straightforward but effective technique to improve abstractive summaries by fine-tuning pre-trained language models, and training them with the BRIO paradigm. We build a text summarization dataset for Vietnamese, called VieSum. We perform experiments with abstractive summarization models trained with the BRIO paradigm on the CNNDM and the VieSum datasets. The results show that the models, trained on basic hardware, outperform all existing abstractive summarization models, especially for Vietnamese.