# Big Bidirectional Insertion Representations for Documents

arXiv: https://arxiv.org/abs/1910.13034 · HTML: https://mlsanity.com/paper/1910.13034/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Oct 2019.

## Built on by (most cited first, 1 of 1)

- [On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformers](https://mlsanity.com/paper/2106.01335/) (2021)

## Abstract (arXiv)

The Insertion Transformer is well suited for long form text generation due to its parallel generation capabilities, requiring O(log2 n) generation steps to generate n tokens. However, modeling long sequences is difficult, as there is more ambiguity captured in the attention mechanism. This work proposes the Big Bidirectional Insertion Representations for Documents (Big BIRD), an insertion-based model for document-level translation tasks. We scale up the insertion-based models to long form documents. Our key contribution is introducing sentence alignment via sentence-positional embeddings between the source and target document. We show an improvement of +4.3 BLEU on the WMT'19 EnglishrightarrowGerman document-level translation task compared with the Insertion Transformer baseline.