# Hard Non-Monotonic Attention for Character-Level Transduction

arXiv: https://arxiv.org/abs/1808.10024 · HTML: https://mlsanity.com/paper/1808.10024/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Aug 2018.

## Built on by (most cited first, 12 of 12)

- [Sequence-to-Sequence Learning with Latent Neural Grammars](https://mlsanity.com/paper/2109.01135/) (2021)
- [Can a Transformer Pass the Wug Test? Tuning Copying Bias in Neural Morphological Inflection Models](https://mlsanity.com/paper/2104.06483/) (2021)
- [Inducing and Using Alignments for Transition-based AMR Parsing](https://mlsanity.com/paper/2205.01464/) (2022)
- [A study of latent monotonic attention variants](https://mlsanity.com/paper/2103.16710/) (2021)
- [On Biasing Transformer Attention Towards Monotonicity](https://mlsanity.com/paper/2104.03945/) (2021)
- [Equivariant Transduction through Invariant Alignment](https://mlsanity.com/paper/2209.10926/) (2022)
- [Computational Morphology with Neural Network Approaches](https://mlsanity.com/paper/2105.09404/) (2021)
- [On the Learning Dynamics of Attention Networks](https://mlsanity.com/paper/2307.13421/) (2023)
- [Comparative Error Analysis in Neural and Finite-state Models for Unsupervised Character-level Transduction](https://mlsanity.com/paper/2106.12698/) (2021)
- [Searching for More Efficient Dynamic Programs](https://mlsanity.com/paper/2109.06966/) (2021)
- [Differentiable Generative Phonology](https://mlsanity.com/paper/2102.05717/) (2021)
- [Cross-lingual Inflection as a Data Augmentation Method for Parsing](https://mlsanity.com/paper/2205.09350/) (2022)

## Abstract (arXiv)

Character-level string-to-string transduction is an important component of various NLP tasks. The goal is to map an input string to an output string, where the strings may be of different lengths and have characters taken from different alphabets. Recent approaches have used sequence-to-sequence models with an attention mechanism to learn which parts of the input string the model should focus on during the generation of the output string. Both soft attention and hard monotonic attention have been used, but hard non-monotonic attention has only been used in other sequence modeling tasks such as image captioning (Xu et al., 2015), and has required a stochastic approximation to compute the gradient. In this work, we introduce an exact, polynomial-time algorithm for marginalizing over the exponential number of non-monotonic alignments between two strings, showing that hard attention models can be viewed as neural reparameterizations of the classical IBM Model 1. We compare soft and hard non-monotonic attention experimentally and find that the exact algorithm significantly improves performance over the stochastic approximation and outperforms soft attention. Code is available at https://github. com/shijie-wu/neural-transducer.