# A Neural Multi-sequence Alignment TeCHnique (NeuMATCH)

arXiv: https://arxiv.org/abs/1803.00057 · HTML: https://mlsanity.com/paper/1803.00057/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 19 Feb 2018.

## Built on by (most cited first, 13 of 13)

- [Learning to Align Sequential Actions in the Wild](https://mlsanity.com/paper/2111.09301/) (2021)
- [Event Causality Is Key to Computational Story Understanding](https://mlsanity.com/paper/2311.09648/) (2023)
- [Aligning Step-by-Step Instructional Diagrams to Video Demonstrations](https://mlsanity.com/paper/2303.13800/) (2023)
- [Movie101v2: Improved Movie Narration Benchmark](https://mlsanity.com/paper/2404.13370/) (2024)
- [Synopses of Movie Narratives: a Video-Language Dataset for Story Understanding](https://mlsanity.com/paper/2203.05711/) (2022)
- [A Survey of Computer Vision Technologies In Urban and Controlled-environment Agriculture](https://mlsanity.com/paper/2210.11318/) (2022)
- [A Convolutional-Attentional Neural Framework for Structure-Aware Performance-Score Synchronization](https://mlsanity.com/paper/2204.08822/) (2022)
- [PTVD: A Large-Scale Plot-Oriented Multimodal Dataset Based on Television Dramas](https://mlsanity.com/paper/2306.14644/) (2023)
- [ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment](https://mlsanity.com/paper/2506.22967/) (2025)
- [Multilingual Synopses of Movie Narratives: A Dataset for Vision-Language Story Understanding](https://mlsanity.com/paper/2406.13092/) (2024)
- [Two Causally Related Needles in a Video Haystack](https://mlsanity.com/paper/2505.19853/) (2025)
- [Aligning Moments in Time using Video Queries](https://mlsanity.com/paper/2508.15439/) (2025)
- [REFRAMED: Towards Realistic Audio Description Generation for Movies](https://mlsanity.com/paper/2608.09765/) (2026)

## Abstract (arXiv)

The alignment of heterogeneous sequential data (video to text) is an important and challenging problem. Standard techniques for this task, including Dynamic Time Warping (DTW) and Conditional Random Fields (CRFs), suffer from inherent drawbacks. Mainly, the Markov assumption implies that, given the immediate past, future alignment decisions are independent of further history. The separation between similarity computation and alignment decision also prevents end-to-end training. In this paper, we propose an end-to-end neural architecture where alignment actions are implemented as moving data between stacks of Long Short-term Memory (LSTM) blocks. This flexible architecture supports a large variety of alignment tasks, including one-to-one, one-to-many, skipping unmatched elements, and (with extensions) non-monotonic alignment. Extensive experiments on semi-synthetic and real datasets show that our algorithm outperforms state-of-the-art baselines.