# Countering the Effects of Lead Bias in News Summarization via Multi-Stage Training and Auxiliary Losses

arXiv: https://arxiv.org/abs/1909.04028 · HTML: https://mlsanity.com/paper/1909.04028/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 8 Sep 2019.

## Built on by (most cited first, 22 of 22)

- [An Empirical Survey on Long Document Summarization: Datasets, Models and Metrics](https://mlsanity.com/paper/2207.00939/) (2022)
- [Does Pretraining for Summarization Require Knowledge Transfer?](https://mlsanity.com/paper/2109.04953/) (2021)
- [Revisiting Zero-Shot Abstractive Summarization in the Era of Large Language Models from the Perspective of Position Bias](https://mlsanity.com/paper/2401.01989/) (2024)
- [SEMQA: Semi-Extractive Multi-Source Question Answering](https://mlsanity.com/paper/2311.04886/) (2023)
- [StreamHover: Livestream Transcript Summarization and Annotation](https://mlsanity.com/paper/2109.05160/) (2021)
- [A Sliding-Window Approach to Automatic Creation of Meeting Minutes](https://mlsanity.com/paper/2104.12324/) (2021)
- [Demoting the Lead Bias in News Summarization via Alternating Adversarial Learning](https://mlsanity.com/paper/2105.14241/) (2021)
- [TLDR9+: A Large Scale Resource for Extreme Summarization of Social Media Posts](https://mlsanity.com/paper/2110.01159/) (2021)
- [CD2CR: Co-reference Resolution Across Documents and Domains](https://mlsanity.com/paper/2101.12637/) (2021)
- [Understanding Position Bias Effects on Fairness in Social Multi-Document Summarization](https://mlsanity.com/paper/2405.01790/) (2024)
- [Leveraging Non-dialogue Summaries for Dialogue Summarization](https://mlsanity.com/paper/2210.09474/) (2022)
- [Document Summarization with Text Segmentation](https://mlsanity.com/paper/2301.08817/) (2023)
- [MiRANews: Dataset and Benchmarks for Multi-Resource-Assisted News Summarization](https://mlsanity.com/paper/2109.10650/) (2021)
- [Modeling Endorsement for Multi-Document Abstractive Summarization](https://mlsanity.com/paper/2110.07844/) (2021)
- [Unsupervised Abstractive Dialogue Summarization with Word Graphs and POV Conversion](https://mlsanity.com/paper/2205.13108/) (2022)
- [Attacks against Abstractive Text Summarization Models through Lead Bias and Influence Functions](https://mlsanity.com/paper/2410.20019/) (2024)
- [Measuring Alignment With Reader Highlights Net of Position and Length](https://mlsanity.com/paper/2607.27739/) (2026)
- [Text Summarization with Oracle Expectation](https://mlsanity.com/paper/2209.12714/) (2022)
- [Self-Supervised Position Debiasing for Large Language Models](https://mlsanity.com/paper/2401.01218/) (2024)
- [Detect, Unlearn, Restore: Defending Text Summarization Models Against Data Poisoning](https://mlsanity.com/paper/2606.26036/) (2026)
- [DiscoSum: Discourse-aware News Summarization](https://mlsanity.com/paper/2506.06930/) (2025)
- [A Systematic Evaluation of Positional Bias in Multi-Video Summarization with MLLMs](https://mlsanity.com/paper/2606.04596/) (2026)

## Abstract (arXiv)

Sentence position is a strong feature for news summarization, since the lead often (but not always) summarizes the key points of the article. In this paper, we show that recent neural systems excessively exploit this trend, which although powerful for many inputs, is also detrimental when summarizing documents where important content should be extracted from later parts of the article. We propose two techniques to make systems sensitive to the importance of content in different parts of the article. The first technique employs 'unbiased' data; i.e., randomly shuffled sentences of the source document, to pretrain the model. The second technique uses an auxiliary ROUGE-based loss that encourages the model to distribute importance scores throughout a document by mimicking sentence-level ROUGE scores on the training data. We show that these techniques significantly improve the performance of a competitive reinforcement learning based extractive system, with the auxiliary loss being more powerful than pretraining.