# REMAP: Regularized Matching and Partial Alignment of Video Embeddings

arXiv: https://arxiv.org/abs/2509.24382 · HTML: https://mlsanity.com/paper/2509.24382/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 43 of its 74 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Builds on (most cited first, 30 of 43)

- [Quo Vadis, Action Recognition? A New Model and the Kinetics Dataset](https://mlsanity.com/paper/1705.07750/) (2017)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Unsupervised Representation Learning by Predicting Image Rotations](https://mlsanity.com/paper/1803.07728/) (2018)
- [Two-Stream Convolutional Networks for Action Recognition in Videos](https://mlsanity.com/paper/1406.2199/) (2014)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Towards Automatic Learning of Procedures from Web Instructional Videos](https://mlsanity.com/paper/1703.09788/) (2017)
- [Domain Generalization by Solving Jigsaw Puzzles](https://mlsanity.com/paper/1903.06864/) (2019)
- [Unsupervised Learning of Video Representations using LSTMs](https://mlsanity.com/paper/1502.04681/) (2015)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Unsupervised Representation Learning by Sorting Sequences](https://mlsanity.com/paper/1708.01246/) (2017)
- [Cross-task weakly supervised learning from instructional videos](https://mlsanity.com/paper/1903.08225/) (2019)
- [Video Representation Learning by Dense Predictive Coding](https://mlsanity.com/paper/1909.04656/) (2019)
- [SpeedNet: Learning the Speediness in Videos](https://mlsanity.com/paper/2004.06130/) (2020)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [Self-Supervised Video Representation Learning with Space-Time Cubic Puzzles](https://mlsanity.com/paper/1811.09795/) (2018)
- [Self-supervised Video Representation Learning by Pace Prediction](https://mlsanity.com/paper/2008.05861/) (2020)
- [Unsupervised Learning from Narrated Instruction Videos](https://mlsanity.com/paper/1506.09215/) (2015)
- [Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learning](https://mlsanity.com/paper/2006.11476/) (2020)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [My View is the Best View: Procedure Learning from Egocentric Videos](https://mlsanity.com/paper/2207.10883/) (2022)
- [The MECCANO Dataset: Understanding Human-Object Interactions from Egocentric Videos in an Industrial-like Domain](https://mlsanity.com/paper/2010.05654/) (2020)
- [Unsupervised learning of action classes with continuous temporal embedding](https://mlsanity.com/paper/1904.04189/) (2019)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [Learning Image Representations by Completing Damaged Jigsaw Puzzles](https://mlsanity.com/paper/1802.01880/) (2018)
- [Unsupervised Action Segmentation by Joint Representation Learning and Online Clustering](https://mlsanity.com/paper/2105.13353/) (2021)
- [DynamoNet: Dynamic Action and Motion Network](https://mlsanity.com/paper/1904.11407/) (2019)
- [Learning by Aligning Videos in Time](https://mlsanity.com/paper/2103.17260/) (2021)
- [Joint Visual-Temporal Embedding for Unsupervised Learning of Actions in Untrimmed Sequences](https://mlsanity.com/paper/2001.11122/) (2020)
- [Representation Learning via Global Temporal Alignment and Cycle-Consistency](https://mlsanity.com/paper/2105.05217/) (2021)

## Built on by (most cited first, 1 of 1)

- [Loss Knows Best: Detecting Annotation Errors in Videos via Loss Trajectories](https://mlsanity.com/paper/2602.15154/) (2026)

## Abstract (arXiv)

Real-world instructional videos are long, noisy, and often contain extended background segments, repeated actions, and execution variability that do not correspond to meaningful procedural steps. We propose **REMAP**, an unsupervised framework for procedure learning based on *Regularized Fused Partial Gromov-Wasserstein Optimal Transport*. REMAP relaxes balanced transport constraints, allowing non-informative or redundant frames to remain unmatched through partial transport. The formulation jointly models semantic similarity and temporal structure, while incorporating Laplacian-based smoothness and structural regularization to prevent degenerate alignments and reduce background interference. We evaluate REMAP on large-scale egocentric and third-person benchmarks. The method consistently outperforms state-of-the-art approaches, achieving up to **11.6% (+4.45pp)** F1 and **19.6% (+4.73pp)** IoU improvements on EgoProceL, and an average **41% (+17.15pp)** F1 gain on ProceL and CrossTask. These results highlight the importance of partial alignment in handling real-world procedural variability and demonstrate that REMAP provides a robust and scalable approach for instructional video understanding.