# BioCopy: A Plug-And-Play Span Copy Mechanism in Seq2Seq Models

arXiv: https://arxiv.org/abs/2109.12533 · HTML: https://mlsanity.com/paper/2109.12533/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 19 references that are in the map.
- First posted on arXiv 26 Sep 2021.

## Builds on (most cited first, 15 of 15)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Pointer Networks](https://mlsanity.com/paper/1506.03134/) (2015)
- [Unified Language Model Pre-training for Natural Language Understanding and Generation](https://mlsanity.com/paper/1905.03197/) (2019)
- [Highway Networks](https://mlsanity.com/paper/1505.00387/) (2015)
- [Joint Extraction of Entities and Relations Based on a Novel Tagging Scheme](https://mlsanity.com/paper/1706.05075/) (2017)
- [Heterogeneous Graph Neural Networks for Extractive Document Summarization](https://mlsanity.com/paper/2004.12393/) (2020)
- [Effective Modeling of Encoder-Decoder Architecture for Joint Entity and Relation Extraction](https://mlsanity.com/paper/1911.09886/) (2019)
- [Addressing the Rare Word Problem in Neural Machine Translation](https://mlsanity.com/paper/1410.8206/) (2014)
- [CopyMTL: Copy Mechanism for Joint Extraction of Entities and Relations with Multi-Task Learning](https://mlsanity.com/paper/1911.10438/) (2019)
- [Pointing the Unknown Words](https://mlsanity.com/paper/1603.08148/) (2016)
- [A Closer Look at Data Bias in Neural Extractive Summarization Models](https://mlsanity.com/paper/1909.13705/) (2019)
- [Exploring Domain Shift in Extractive Text Summarization](https://mlsanity.com/paper/1908.11664/) (2019)

## Built on by (most cited first, 2 of 2)

- [Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistency](https://mlsanity.com/paper/2209.03479/) (2022)
- [A Unified Generative Approach to Product Attribute-Value Identification](https://mlsanity.com/paper/2306.05605/) (2023)

## Abstract (arXiv)

Copy mechanisms explicitly obtain unchanged tokens from the source (input) sequence to generate the target (output) sequence under the neural seq2seq framework. However, most of the existing copy mechanisms only consider single word copying from the source sentences, which results in losing essential tokens while copying long spans. In this work, we propose a plug-and-play architecture, namely BioCopy, to alleviate the problem aforementioned. Specifically, in the training stage, we construct a BIO tag for each token and train the original model with BIO tags jointly. In the inference stage, the model will firstly predict the BIO tag at each time step, then conduct different mask strategies based on the predicted BIO label to diminish the scope of the probability distributions over the vocabulary list. Experimental results on two separate generative tasks show that they all outperform the baseline models by adding our BioCopy to the original model structure.