# Attending to Future Tokens For Bidirectional Sequence Generation

arXiv: https://arxiv.org/abs/1908.05915 · HTML: https://mlsanity.com/paper/1908.05915/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 16 Aug 2019.

## Built on by (most cited first, 12 of 12)

- [PIXAR: Auto-Regressive Language Modeling in Pixel Space](https://mlsanity.com/paper/2401.03321/) (2024)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2103.03125/) (2021)
- [Open-Retrieval Conversational Machine Reading](https://mlsanity.com/paper/2102.08633/) (2021)
- [Smoothing Dialogue States for Open Conversational Machine Reading](https://mlsanity.com/paper/2108.12599/) (2021)
- [Reasoning over Logically Interacted Conditions for Question Answering](https://mlsanity.com/paper/2205.12898/) (2022)
- [ET5: A Novel End-to-end Framework for Conversational Machine Reading Comprehension](https://mlsanity.com/paper/2209.11484/) (2022)
- [Towards End-to-End Open Conversational Machine Reading](https://mlsanity.com/paper/2210.07113/) (2022)
- [Explicit Alignment and Many-to-many Entailment Based Reasoning for Conversational Machine Reading](https://mlsanity.com/paper/2310.13409/) (2023)
- [Advances in Multi-turn Dialogue Comprehension: A Survey](https://mlsanity.com/paper/2110.04984/) (2021)
- [Bridging The Gap: Entailment Fused-T5 for Open-retrieval Conversational Machine Reading Comprehension](https://mlsanity.com/paper/2212.09353/) (2022)
- [A Framework for Bidirectional Decoding: Case Study in Morphological Inflection](https://mlsanity.com/paper/2305.12580/) (2023)
- [Machine Learning Predictors for Min-Entropy Estimation](https://mlsanity.com/paper/2406.19983/) (2024)

## Abstract (arXiv)

Neural sequence generation is typically performed token-by-token and left-to-right. Whenever a token is generated only previously produced tokens are taken into consideration. In contrast, for problems such as sequence classification, bidirectional attention, which takes both past and future tokens into consideration, has been shown to perform much better. We propose to make the sequence generation process bidirectional by employing special placeholder tokens. Treated as a node in a fully connected graph, a placeholder token can take past and future tokens into consideration when generating the actual output token. We verify the effectiveness of our approach experimentally on two conversational tasks where the proposed bidirectional model outperforms competitive baselines by a large margin.