# Uncertainty-DTW for Time Series and Sequences

arXiv: https://arxiv.org/abs/2211.00005 · HTML: https://mlsanity.com/paper/2211.00005/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 61 references that are in the map.
- First posted on arXiv 30 Oct 2022.

## Builds on (most cited first, 30 of 37)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [The Kinetics Human Action Video Dataset](https://mlsanity.com/paper/1705.06950/) (2017)
- [Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition](https://mlsanity.com/paper/1801.07455/) (2018)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [NTU RGB+D: A Large Scale Dataset for 3D Human Activity Analysis](https://mlsanity.com/paper/1604.02808/) (2016)
- [NTU RGB+D 120: A Large-Scale Benchmark for 3D Human Activity Understanding](https://mlsanity.com/paper/1905.04757/) (2019)
- [A simple yet effective baseline for 3d human pose estimation](https://mlsanity.com/paper/1705.03098/) (2017)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [MINIROCKET: A Very Fast (Almost) Deterministic Transform for Time Series Classification](https://mlsanity.com/paper/2012.08791/) (2020)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [Voice2Series: Reprogramming Acoustic Models for Time Series Classification](https://mlsanity.com/paper/2106.09296/) (2021)
- [Temporal-Relational CrossTransformers for Few-Shot Action Recognition](https://mlsanity.com/paper/2101.06184/) (2021)
- [End-to-End Adversarial Text-to-Speech](https://mlsanity.com/paper/2006.03575/) (2020)
- [TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition](https://mlsanity.com/paper/1907.09021/) (2019)
- [Differentiable Dynamic Programming for Structured Prediction and Attention](https://mlsanity.com/paper/1802.03676/) (2018)
- [Unsupervised Pretraining for Sequence to Sequence Learning](https://mlsanity.com/paper/1611.02683/) (2016)
- [COSTA: Covariance-Preserving Feature Augmentation for Graph Contrastive Learning](https://mlsanity.com/paper/2206.04726/) (2022)
- [A Comparative Review of Recent Kinect-based Action Recognition Algorithms](https://mlsanity.com/paper/1906.09955/) (2019)
- [Differentiable Divergences Between Time Series](https://mlsanity.com/paper/2010.08354/) (2020)
- [Tensor Representations for Action Recognition](https://mlsanity.com/paper/2012.14371/) (2020)
- [Self-supervising Action Recognition by Statistical Moment and Subspace Descriptors](https://mlsanity.com/paper/2001.04627/) (2020)
- [Contrastive Laplacian Eigenmaps](https://mlsanity.com/paper/2201.05493/) (2022)
- [Skeleton-DML: Deep Metric Learning for Skeleton-Based One-Shot Action Recognition](https://mlsanity.com/paper/2012.13823/) (2020)
- [Temporal Transformer Networks: Joint Learning of Invariant and Discriminative Time Warping](https://mlsanity.com/paper/1906.05947/) (2019)
- [Fisher-Bures Adversary Graph Convolutional Networks](https://mlsanity.com/paper/1903.04154/) (2019)
- [SL-DML: Signal Level Deep Metric Learning for Multimodal One-Shot Action Recognition](https://mlsanity.com/paper/2004.11085/) (2020)
- [HOPC: Histogram of Oriented Principal Components of 3D Pointclouds for Action Recognition](https://mlsanity.com/paper/1408.3809/) (2014)

## Built on by (most cited first, 21 of 21)

- [Advancing Video Anomaly Detection: A Concise Review and a New Dataset](https://mlsanity.com/paper/2402.04857/) (2024)
- [3Mformer: Multi-order Multi-mode Transformer for Skeletal Action Recognition](https://mlsanity.com/paper/2303.14474/) (2023)
- [Transductive Few-shot Learning with Prototype-based Label Propagation by Iterative Graph Refinement](https://mlsanity.com/paper/2304.11598/) (2023)
- [Distilling Self-Supervised Vision Transformers for Weakly-Supervised Few-Shot Classification & Segmentation](https://mlsanity.com/paper/2307.03407/) (2023)
- [Learnable Expansion of Graph Operators for Multi-Modal Feature Fusion](https://mlsanity.com/paper/2410.01506/) (2024)
- [Do Language Models Understand Time?](https://mlsanity.com/paper/2412.13845/) (2024)
- [One-Shot Action Recognition via Multi-Scale Spatial-Temporal Skeleton Matching](https://mlsanity.com/paper/2307.07286/) (2023)
- [Flow Dynamics Correction for Action Recognition](https://mlsanity.com/paper/2310.10059/) (2023)
- [Traffic Forecasting on New Roads Using Spatial Contrastive Pre-Training (SCPT)](https://mlsanity.com/paper/2305.05237/) (2023)
- [CrossGLG: LLM Guides One-shot Skeleton-based 3D Action Recognition in a Cross-level Manner](https://mlsanity.com/paper/2403.10082/) (2024)
- [Meet JEANIE: a Similarity Measure for 3D Skeleton Sequences via Temporal-Viewpoint Alignment](https://mlsanity.com/paper/2402.04599/) (2024)
- [Adaptive Multi-head Contrastive Learning](https://mlsanity.com/paper/2310.05615/) (2023)
- [MoFM: A Large-Scale Human Motion Foundation Model](https://mlsanity.com/paper/2502.05432/) (2025)
- [SkeletonX: Data-Efficient Skeleton-based Action Recognition via Cross-sample Feature Aggregation](https://mlsanity.com/paper/2504.11749/) (2025)
- [Feature Hallucination for Self-supervised Action Recognition](https://mlsanity.com/paper/2506.20342/) (2025)
- [Exploiting Field Dependencies for Learning on Categorical Data](https://mlsanity.com/paper/2307.09321/) (2023)
- [Recovering Complete Actions for Cross-dataset Skeleton Action Recognition](https://mlsanity.com/paper/2410.23641/) (2024)
- [Learning Time in Static Classifiers](https://mlsanity.com/paper/2511.12321/) (2025)
- [Evolving Skeletons: Motion Dynamics in Action Recognition](https://mlsanity.com/paper/2501.02593/) (2025)
- [Spectral-Aware Text-to-Time Series Generation with Billion-Scale Multimodal Meteorological Data](https://mlsanity.com/paper/2603.27135/) (2026)
- [Subspace Kernel Learning on Tensor Sequences](https://mlsanity.com/paper/2603.19546/) (2026)

## Abstract (arXiv)

Dynamic Time Warping (DTW) is used for matching pairs of sequences and celebrated in applications such as forecasting the evolution of time series, clustering time series or even matching sequence pairs in few-shot action recognition. The transportation plan of DTW contains a set of paths; each path matches frames between two sequences under a varying degree of time warping, to account for varying temporal intra-class dynamics of actions. However, as DTW is the smallest distance among all paths, it may be affected by the feature uncertainty which varies across time steps/frames. Thus, in this paper, we propose to model the so-called aleatoric uncertainty of a differentiable (soft) version of DTW. To this end, we model the heteroscedastic aleatoric uncertainty of each path by the product of likelihoods from Normal distributions, each capturing variance of pair of frames. (The path distance is the sum of base distances between features of pairs of frames of the path.) The Maximum Likelihood Estimation (MLE) applied to a path yields two terms: (i) a sum of Euclidean distances weighted by the variance inverse, and (ii) a sum of log-variance regularization terms. Thus, our uncertainty-DTW is the smallest weighted path distance among all paths, and the regularization term (penalty for the high uncertainty) is the aggregate of log-variances along the path. The distance and the regularization term can be used in various objectives. We showcase forecasting the evolution of time series, estimating the Fr'echet mean of time series, and supervised/unsupervised few-shot action recognition of the articulated human 3D body joints.