# Learning Discriminative Prototypes with Dynamic Time Warping

arXiv: https://arxiv.org/abs/2103.09458 · HTML: https://mlsanity.com/paper/2103.09458/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 38 references that are in the map.
- First posted on arXiv 17 Mar 2021.

## Builds on (most cited first, 19 of 19)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [The UCR Time Series Archive](https://mlsanity.com/paper/1810.07758/) (2018)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Deep Reinforcement Learning for Unsupervised Video Summarization with Diversity-Representativeness Reward](https://mlsanity.com/paper/1801.00054/) (2017)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Weakly-Supervised Action Segmentation with Iterative Soft Boundary Assignment](https://mlsanity.com/paper/1803.10699/) (2018)
- [Connectionist Temporal Modeling for Weakly Supervised Action Labeling](https://mlsanity.com/paper/1607.08584/) (2016)
- [Weakly Supervised Action Learning with RNN based Fine-to-coarse Modeling](https://mlsanity.com/paper/1703.08132/) (2017)
- [Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models](https://mlsanity.com/paper/1909.09020/) (2019)
- [Summarizing Videos with Attention](https://mlsanity.com/paper/1812.01969/) (2018)
- [NeuralNetwork-Viterbi: A Framework for Weakly Supervised Video Learning](https://mlsanity.com/paper/1805.06875/) (2018)
- [Differentiable Dynamic Programming for Structured Prediction and Attention](https://mlsanity.com/paper/1802.03676/) (2018)
- [Weakly Supervised Energy-Based Learning for Action Segmentation](https://mlsanity.com/paper/1909.13155/) (2019)
- [Weakly Supervised Action Labeling in Videos Under Ordering Constraints](https://mlsanity.com/paper/1407.1208/) (2014)
- [shapeDTW: shape Dynamic Time Warping](https://mlsanity.com/paper/1606.01601/) (2016)
- [Temporal Transformer Networks: Joint Learning of Invariant and Discriminative Time Warping](https://mlsanity.com/paper/1906.05947/) (2019)
- [Weakly Supervised Video Summarization by Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2001.05864/) (2020)
- [Comprehensive Video Understanding: Video summarization with content-based video recommender design](https://mlsanity.com/paper/1910.13888/) (2019)
- [ILS-SUMM: Iterated Local Search for Unsupervised Video Summarization](https://mlsanity.com/paper/1912.03650/) (2019)

## Built on by (most cited first, 23 of 23)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Temporal Alignment Networks for Long-term Video](https://mlsanity.com/paper/2204.02968/) (2022)
- [Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers](https://mlsanity.com/paper/2108.11996/) (2021)
- [StepFormer: Self-supervised Step Discovery and Localization in Instructional Videos](https://mlsanity.com/paper/2304.13265/) (2023)
- [Fine-grained Temporal Contrastive Learning for Weakly-supervised Temporal Action Localization](https://mlsanity.com/paper/2203.16800/) (2022)
- [Video-Text Representation Learning via Differentiable Weak Temporal Alignment](https://mlsanity.com/paper/2203.16784/) (2022)
- [Efficient and Effective Weakly-Supervised Action Segmentation via Action-Transition-Aware Boundary Alignment](https://mlsanity.com/paper/2403.19225/) (2024)
- [Causal Temporal Representation Learning with Nonstationary Sparse Transition](https://mlsanity.com/paper/2409.03142/) (2024)
- [Weakly-Supervised Online Action Segmentation in Multi-View Instructional Videos](https://mlsanity.com/paper/2203.13309/) (2022)
- [Hierarchical Modeling for Task Recognition and Action Segmentation in Weakly-Labeled Instructional Videos](https://mlsanity.com/paper/2110.05697/) (2021)
- [Deep Learning Approaches for Seizure Video Analysis: A Review](https://mlsanity.com/paper/2312.10930/) (2023)
- [CLOT: Closed Loop Optimal Transport for Unsupervised Action Segmentation](https://mlsanity.com/paper/2507.03539/) (2025)
- [Robust Time Series Dissimilarity Measure for Outlier Detection and Periodicity Detection](https://mlsanity.com/paper/2206.02956/) (2022)
- [Deep Declarative Dynamic Time Warping for End-to-End Learning of Alignment Paths](https://mlsanity.com/paper/2303.10778/) (2023)
- [Transformers in Action: Weakly Supervised Action Segmentation](https://mlsanity.com/paper/2201.05675/) (2022)
- [Graph2Vid: Flow graph to Video Grounding for Weakly-supervised Multi-Step Localization](https://mlsanity.com/paper/2210.04996/) (2022)
- [2by2: Weakly-Supervised Learning for Global Action Segmentation](https://mlsanity.com/paper/2412.12829/) (2024)
- [Unlabeled Action Quality Assessment Based on Multi-dimensional Adaptive Constrained Dynamic Time Warping](https://mlsanity.com/paper/2410.14161/) (2024)
- [Pose-Aware Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2504.05700/) (2025)
- [Approximating DTW with a convolutional neural network on EEG data](https://mlsanity.com/paper/2301.12873/) (2023)
- [TimewarpVAE: Simultaneous Time-Warping and Representation Learning of Trajectories](https://mlsanity.com/paper/2310.16027/) (2023)
- [Hierarchical Action Learning for Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2602.24275/) (2026)
- [Boundary-Centric Clip-Budgeted Active Learning for Temporal Action Segmentation](https://mlsanity.com/paper/2604.15173/) (2026)

## Abstract (arXiv)

Dynamic Time Warping (DTW) is widely used for temporal data processing. However, existing methods can neither learn the discriminative prototypes of different classes nor exploit such prototypes for further analysis. We propose Discriminative Prototype DTW (DP-DTW), a novel method to learn class-specific discriminative prototypes for temporal recognition tasks. DP-DTW shows superior performance compared to conventional DTWs on time series classification benchmarks. Combined with end-to-end deep learning, DP-DTW can handle challenging weakly supervised action segmentation problems and achieves state of the art results on standard benchmarks. Moreover, detailed reasoning on the input video is enabled by the learned action prototypes. Specifically, an action-based video summarization can be obtained by aligning the input sequence with action prototypes.