# Self-Attentive Model for Headline Generation

arXiv: https://arxiv.org/abs/1901.07786 · HTML: https://mlsanity.com/paper/1901.07786/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2019.

## Built on by (most cited first, 16 of 16)

- [The Russian-focused embedders' exploration: ruMTEB benchmark and Russian embedding model design](https://mlsanity.com/paper/2408.12503/) (2024)
- [NewsBERT: Distilling Pre-trained Language Model for Intelligent News Application](https://mlsanity.com/paper/2102.04887/) (2021)
- ["Why is this misleading?": Detecting News Headline Hallucinations with Explanations](https://mlsanity.com/paper/2302.05852/) (2023)
- [Harnessing the Power of LLMs: Evaluating Human-AI Text Co-Creation through the Lens of News Headline Generation](https://mlsanity.com/paper/2310.10706/) (2023)
- [Fine-tuning GPT-3 for Russian Text Summarization](https://mlsanity.com/paper/2108.03502/) (2021)
- [Large Scale Multi-Lingual Multi-Modal Summarization Dataset](https://mlsanity.com/paper/2302.06560/) (2023)
- [Follow the Timeline! Generating Abstractive and Extractive Timeline Summary in Chronological Order](https://mlsanity.com/paper/2301.00867/) (2023)
- [Contrastive Learning enhanced Author-Style Headline Generation](https://mlsanity.com/paper/2211.03305/) (2022)
- [Let AI Entertain You: Increasing User Engagement with Generative AI and Rejection Sampling](https://mlsanity.com/paper/2312.12457/) (2023)
- [Fact-Preserved Personalized News Headline Generation](https://mlsanity.com/paper/2501.11828/) (2025)
- [Building Russian Benchmark for Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2504.12879/) (2025)
- [BeliN: A Novel Corpus for Bengali Religious News Headline Generation using Contextual Feature Fusion](https://mlsanity.com/paper/2501.01069/) (2025)
- [Introducing Spotlight: A Novel Approach for Generating Captivating Key Information from Documents](https://mlsanity.com/paper/2509.10935/) (2025)
- [COBART: Controlled, Optimized, Bidirectional and Auto-Regressive Transformer for Ad Headline Generation](https://mlsanity.com/paper/2607.08071/) (2026)
- [Improving Sequence-to-Sequence Models for Abstractive Text Summarization Using Meta Heuristic Approaches](https://mlsanity.com/paper/2403.16247/) (2024)
- [KoWit-24: A Richly Annotated Dataset of Wordplay in News Headlines](https://mlsanity.com/paper/2503.01510/) (2025)

## Abstract (arXiv)

Headline generation is a special type of text summarization task. While the amount of available training data for this task is almost unlimited, it still remains challenging, as learning to generate headlines for news articles implies that the model has strong reasoning about natural language. To overcome this issue, we applied recent Universal Transformer architecture paired with byte-pair encoding technique and achieved new state-of-the-art results on the New York Times Annotated corpus with ROUGE-L F1-score 24.84 and ROUGE-2 F1-score 13.48. We also present the new RIA corpus and reach ROUGE-L F1-score 36.81 and ROUGE-2 F1-score 22.15 on it.