# Leveraging Key Information Modeling to Improve Less-Data Constrained News Headline Generation via Duality Fine-Tuning

arXiv: https://arxiv.org/abs/2210.04473 · HTML: https://mlsanity.com/paper/2210.04473/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 45 references that are in the map.
- First posted on arXiv 10 Oct 2022.

## Builds on (most cited first, 27 of 27)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Text Summarization with Pretrained Encoders](https://mlsanity.com/paper/1908.08345/) (2019)
- [Convolutional Sequence to Sequence Learning](https://mlsanity.com/paper/1705.03122/) (2017)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [DualGAN: Unsupervised Dual Learning for Image-to-Image Translation](https://mlsanity.com/paper/1704.02510/) (2017)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Dual Learning for Machine Translation](https://mlsanity.com/paper/1611.00179/) (2016)
- [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021)
- [Improving Factual Consistency of Abstractive Summarization via Question Answering](https://mlsanity.com/paper/2105.04623/) (2021)
- [AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarization](https://mlsanity.com/paper/2103.11332/) (2021)
- [Pretraining-Based Natural Language Generation for Text Summarization](https://mlsanity.com/paper/1902.09243/) (2019)
- [Generating Representative Headlines for News Stories](https://mlsanity.com/paper/2001.09386/) (2020)
- [Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generation](https://mlsanity.com/paper/2010.12723/) (2020)
- [Long Document Summarization in a Low Resource Setting using Pretrained Language Models](https://mlsanity.com/paper/2103.00751/) (2021)
- [Improving Truthfulness of Headline Generation](https://mlsanity.com/paper/2005.00882/) (2020)
- [Dual Supervised Learning for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.06196/) (2019)
- [Exploring Multitask Learning for Low-Resource AbstractiveSummarization](https://mlsanity.com/paper/2109.08565/) (2021)
- [Dual Inference for Improving Language Understanding and Generation](https://mlsanity.com/paper/2010.04246/) (2020)
- [Transformer-based Lexically Constrained Headline Generation](https://mlsanity.com/paper/2109.07080/) (2021)

## Built on by (most cited first, 1 of 1)

- [Grafting Pre-trained Models for Multimodal Headline Generation](https://mlsanity.com/paper/2211.07210/) (2022)

## Abstract (arXiv)

Recent language generative models are mostly trained on large-scale datasets, while in some real scenarios, the training datasets are often expensive to obtain and would be small-scale. In this paper we investigate the challenging task of less-data constrained generation, especially when the generated news headlines are short yet expected by readers to keep readable and informative simultaneously. We highlight the key information modeling task and propose a novel duality fine-tuning method by formally defining the probabilistic duality constraints between key information prediction and headline generation tasks. The proposed method can capture more information from limited data, build connections between separate tasks, and is suitable for less-data constrained generation tasks. Furthermore, the method can leverage various pre-trained generative regimes, e.g., autoregressive and encoder-decoder models. We conduct extensive experiments to demonstrate that our method is effective and efficient to achieve improved performance in terms of language modeling metric and informativeness correctness metric on two public datasets.