# News Headline Grouping as a Challenging NLU Task

arXiv: https://arxiv.org/abs/2105.05391 · HTML: https://mlsanity.com/paper/2105.05391/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 40 references that are in the map.
- First posted on arXiv 12 May 2021.

## Builds on (most cited first, 11 of 11)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Know What You Don't Know: Unanswerable Questions for SQuAD](https://mlsanity.com/paper/1806.03822/) (2018)
- [SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluation](https://mlsanity.com/paper/1708.00055/) (2017)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [Cycle-Consistency for Robust Visual Question Answering](https://mlsanity.com/paper/1902.05660/) (2019)
- [Hooks in the Headline: Learning to Generate Headlines with Controlled Styles](https://mlsanity.com/paper/2004.01980/) (2020)
- [GoodNewsEveryone: A Corpus of News Headlines Annotated with Emotions, Semantic Roles, and Reader Perception](https://mlsanity.com/paper/1912.03184/) (2019)
- [NSTM: Real-Time Query-Driven News Overview Composition at Bloomberg](https://mlsanity.com/paper/2006.01117/) (2020)
- [Lancaster A at SemEval-2017 Task 5: Evaluation metrics matter: predicting sentiment from financial news headlines](https://mlsanity.com/paper/1705.00571/) (2017)

## Built on by (most cited first, 10 of 10)

- [Discord Questions: A Computational Approach To Diversity Analysis in News Coverage](https://mlsanity.com/paper/2211.05007/) (2022)
- [NewsPod: Automatic and Interactive News Podcasts](https://mlsanity.com/paper/2202.07146/) (2022)
- [HeadlineCause: A Dataset of News Headlines for Detecting Causalities](https://mlsanity.com/paper/2108.12626/) (2021)
- [Designing and Evaluating Interfaces that Highlight News Coverage Diversity Using Discord Questions](https://mlsanity.com/paper/2302.08997/) (2023)
- [Cross-Register Projection for Headline Part of Speech Tagging](https://mlsanity.com/paper/2109.07483/) (2021)
- [Near-Negative Distinction: Giving a Second Life to Human Evaluation Datasets](https://mlsanity.com/paper/2205.06871/) (2022)
- [A Commonsense-Infused Language-Agnostic Learning Framework for Enhancing Prediction of Political Polarity in Multilingual News Headlines](https://mlsanity.com/paper/2212.00298/) (2022)
- [Towards a Fully Unsupervised Framework for Intent Induction in Customer Support Dialogues](https://mlsanity.com/paper/2307.15410/) (2023)
- [Weakly Supervised Headline Dependency Parsing](https://mlsanity.com/paper/2301.10371/) (2023)
- [Improving and Evaluating the Detection of Fragmentation in News Recommendations with the Clustering of News Story Chains](https://mlsanity.com/paper/2309.06192/) (2023)

## Abstract (arXiv)

Recent progress in Natural Language Understanding (NLU) has seen the latest models outperform human performance on many standard tasks. These impressive results have led the community to introspect on dataset limitations, and iterate on more nuanced challenges. In this paper, we introduce the task of HeadLine Grouping (HLG) and a corresponding dataset (HLGD) consisting of 20,056 pairs of news headlines, each labeled with a binary judgement as to whether the pair belongs within the same group. On HLGD, human annotators achieve high performance of around 0.9 F-1, while current state-of-the art Transformer models only reach 0.75 F-1, opening the path for further improvements. We further propose a novel unsupervised Headline Generator Swap model for the task of HeadLine Grouping that achieves within 3 F-1 of the best supervised model. Finally, we analyze high-performing models with consistency tests, and find that models are not consistent in their predictions, revealing modeling limits of current architectures.