# May the Force Be with Your Copy Mechanism: Enhanced Supervised-Copy Method for Natural Language Generation

arXiv: https://arxiv.org/abs/2112.10360 · HTML: https://mlsanity.com/paper/2112.10360/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 44 references that are in the map.
- First posted on arXiv 20 Dec 2021.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Sequence to Sequence Learning with Neural Networks](https://mlsanity.com/paper/1409.3215/) (2014)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyond](https://mlsanity.com/paper/1602.06023/) (2016)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [A Neural Attention Model for Abstractive Sentence Summarization](https://mlsanity.com/paper/1509.00685/) (2015)
- [A Deep Reinforced Model for Abstractive Summarization](https://mlsanity.com/paper/1705.04304/) (2017)
- [MASS: Masked Sequence to Sequence Pre-training for Language Generation](https://mlsanity.com/paper/1905.02450/) (2019)
- [The Natural Language Decathlon: Multitask Learning as Question Answering](https://mlsanity.com/paper/1806.08730/) (2018)
- [Challenges in Data-to-Document Generation](https://mlsanity.com/paper/1707.08052/) (2017)
- [Incorporating Copying Mechanism in Sequence-to-Sequence Learning](https://mlsanity.com/paper/1603.06393/) (2016)
- [Survey of the State of the Art in Natural Language Generation: Core tasks, applications and evaluation](https://mlsanity.com/paper/1703.09902/) (2017)
- [Bottom-Up Abstractive Summarization](https://mlsanity.com/paper/1808.10792/) (2018)
- [Data-to-Text Generation with Content Selection and Planning](https://mlsanity.com/paper/1809.00582/) (2018)
- [Few-Shot NLG with Pre-Trained Language Model](https://mlsanity.com/paper/1904.09521/) (2019)
- [Data-to-text Generation with Entity Modeling](https://mlsanity.com/paper/1906.03221/) (2019)
- [Pointing the Unknown Words](https://mlsanity.com/paper/1603.08148/) (2016)
- [PALM: Pre-training an Autoencoding&Autoregressive Language Model for Context-conditioned Generation](https://mlsanity.com/paper/2004.07159/) (2020)
- [Controlling Hallucinations at Word Level in Data-to-Text Generation](https://mlsanity.com/paper/2102.02810/) (2021)
- [Improving Abstraction in Text Summarization](https://mlsanity.com/paper/1808.07913/) (2018)
- [Language as a Latent Variable: Discrete Generative Models for Sentence Compression](https://mlsanity.com/paper/1609.07317/) (2016)
- [End-to-End Content and Plan Selection for Data-to-Text Generation](https://mlsanity.com/paper/1810.04700/) (2018)
- [A Hierarchical Model for Data-to-Text Generation](https://mlsanity.com/paper/1912.10011/) (2019)

## Built on by (most cited first, 3 of 3)

- [Analysis of Plan-based Retrieval for Grounded Text Generation](https://mlsanity.com/paper/2408.10490/) (2024)
- [Injecting knowledge into language generation: a case study in auto-charting after-visit care instructions from medical dialogue](https://mlsanity.com/paper/2306.03652/) (2023)
- [Structured Language Generation Model: Loss Calibration and Formatted Decoding for Robust Structure Prediction and Knowledge Retrieval](https://mlsanity.com/paper/2402.08971/) (2024)

## Abstract (arXiv)

Recent neural sequence-to-sequence models with a copy mechanism have achieved remarkable progress in various text generation tasks. These models addressed out-of-vocabulary problems and facilitated the generation of rare words. However, the identification of the word which needs to be copied is difficult, as observed by prior copy models, which suffer from incorrect generation and lacking abstractness. In this paper, we propose a novel supervised approach of a copy network that helps the model decide which words need to be copied and which need to be generated. Specifically, we re-define the objective function, which leverages source sequences and target vocabularies as guidance for copying. The experimental results on data-to-text generation and abstractive summarization tasks verify that our approach enhances the copying quality and improves the degree of abstractness.