# Video alignment using unsupervised learning of local and global features

arXiv: https://arxiv.org/abs/2304.06841 · HTML: https://mlsanity.com/paper/2304.06841/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 41 references that are in the map.
- First posted on arXiv 13 Apr 2023.

## Builds on (most cited first, 21 of 21)

- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Simple Online and Realtime Tracking with a Deep Association Metric](https://mlsanity.com/paper/1703.07402/) (2017)
- [Time-Contrastive Networks: Self-Supervised Learning from Video](https://mlsanity.com/paper/1704.06888/) (2017)
- [Temporal Cycle-Consistency Learning](https://mlsanity.com/paper/1904.07846/) (2019)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Learning by Aligning Videos in Time](https://mlsanity.com/paper/2103.17260/) (2021)
- [Representation Learning via Global Temporal Alignment and Cycle-Consistency](https://mlsanity.com/paper/2105.05217/) (2021)
- [Learning Cross-Domain Correspondence for Control with Dynamics Cycle-Consistency](https://mlsanity.com/paper/2012.09811/) (2020)
- [MeTRAbs: Metric-Scale Truncation-Robust Heatmaps for Absolute 3D Human Pose Estimation](https://mlsanity.com/paper/2007.07227/) (2020)
- [TA2N: Two-Stage Action Alignment Network for Few-shot Action Recognition](https://mlsanity.com/paper/2107.04782/) (2021)
- [Learning Implicit Temporal Alignment for Few-shot Video Classification](https://mlsanity.com/paper/2105.04823/) (2021)
- [Weakly-Supervised Alignment of Video With Text](https://mlsanity.com/paper/1505.06027/) (2015)
- [Learning to Align Sequential Actions in the Wild](https://mlsanity.com/paper/2111.09301/) (2021)
- [Backbones-Review: Feature Extraction Networks for Deep Learning and Deep Reinforcement Learning Approaches](https://mlsanity.com/paper/2206.08016/) (2022)
- [Aligning Videos in Space and Time](https://mlsanity.com/paper/2007.04515/) (2020)
- [Am I Done? Predicting Action Progress in Videos](https://mlsanity.com/paper/1705.01781/) (2017)
- [Alignment-guided Temporal Attention for Video Action Recognition](https://mlsanity.com/paper/2210.00132/) (2022)
- [Action Completion: A Temporal Model for Moment Detection](https://mlsanity.com/paper/1805.06749/) (2018)
- [Weakly-supervised Representation Learning for Video Alignment and Analysis](https://mlsanity.com/paper/2302.04064/) (2023)
- [Unsupervised Human Action Detection by Action Matching](https://mlsanity.com/paper/1612.00558/) (2016)

## Built on by (most cited first, 2 of 2)

- [Sync from the Sea: Retrieving Alignable Videos from Large-Scale Datasets](https://mlsanity.com/paper/2409.01445/) (2024)
- [Deep Time Warping for Multiple Time Series Alignment](https://mlsanity.com/paper/2502.16324/) (2025)

## Abstract (arXiv)

In this paper, we tackle the problem of video alignment, the process of matching the frames of a pair of videos containing similar actions. The main challenge in video alignment is that accurate correspondence should be established despite the differences in the execution processes and appearances between the two videos. We introduce an unsupervised method for alignment that uses global and local features of the frames. In particular, we introduce effective features for each video frame by means of three machine vision tools: person detection, pose estimation, and VGG network. Then the features are processed and combined to construct a multidimensional time series that represent the video. The resulting time series are used to align videos of the same actions using a novel version of dynamic time warping named Diagonalized Dynamic Time Warping(DDTW). The main advantage of our approach is that no training is required, which makes it applicable for any new type of action without any need to collect training samples for it. Additionally, our approach can be used for framewise labeling of action phases in a dataset with only a few labeled videos. For evaluation, we considered video synchronization and phase classification tasks on the Penn action and subset of UCF101 datasets. Also, for an effective evaluation of the video synchronization task, we present a new metric called Enclosed Area Error(EAE). The results show that our method outperforms previous state-of-the-art methods, such as TCC, and other self-supervised and weakly supervised methods.