# Deep Declarative Dynamic Time Warping for End-to-End Learning of Alignment Paths

arXiv: https://arxiv.org/abs/2303.10778 · HTML: https://mlsanity.com/paper/2303.10778/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 68 references that are in the map.
- First posted on arXiv 19 Mar 2023.

## Builds on (most cited first, 30 of 36)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [NetVLAD: CNN architecture for weakly supervised place recognition](https://mlsanity.com/paper/1511.07247/) (2015)
- [OptNet: Differentiable Optimization as a Layer in Neural Networks](https://mlsanity.com/paper/1703.00443/) (2017)
- [Differentiable Convex Optimization Layers](https://mlsanity.com/paper/1910.12430/) (2019)
- [Meta-Learning with Differentiable Convex Optimization](https://mlsanity.com/paper/1904.03758/) (2019)
- [Soft-DTW: a Differentiable Loss Function for Time-Series](https://mlsanity.com/paper/1703.01541/) (2017)
- [Enabling Factorized Piano Music Modeling and Generation with the MAESTRO Dataset](https://mlsanity.com/paper/1810.12247/) (2018)
- [Differentiable MPC for End-to-end Planning and Control](https://mlsanity.com/paper/1810.13400/) (2018)
- [Few-Shot Video Classification via Temporal Alignment](https://mlsanity.com/paper/1906.11415/) (2019)
- [VPR-Bench: An Open-Source Visual Place Recognition Evaluation Framework with Quantifiable Viewpoint and Appearance Change](https://mlsanity.com/paper/2005.08135/) (2020)
- [Deep Visual Geo-localization Benchmark](https://mlsanity.com/paper/2204.03444/) (2022)
- [D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation](https://mlsanity.com/paper/1901.02598/) (2019)
- [Where is your place, Visual Place Recognition?](https://mlsanity.com/paper/2103.06443/) (2021)
- [Deep Declarative Networks: A New Hope](https://mlsanity.com/paper/1909.04866/) (2019)
- [Differentiating Through a Cone Program](https://mlsanity.com/paper/1904.09043/) (2019)
- [Shape and Time Distortion Loss for Training Deep Time Series Forecasting Models](https://mlsanity.com/paper/1909.09020/) (2019)
- [End-to-End Learnable Geometric Vision by Backpropagating PnP Optimization](https://mlsanity.com/paper/1909.06043/) (2019)
- [SeqNet: Learning Descriptors for Sequence-based Hierarchical Place Recognition](https://mlsanity.com/paper/2102.11603/) (2021)
- [Learning by Aligning Videos in Time](https://mlsanity.com/paper/2103.17260/) (2021)
- [Drop-DTW: Aligning Common Signal Between Sequences While Dropping Outliers](https://mlsanity.com/paper/2108.11996/) (2021)
- [Solving the Blind Perspective-n-Point Problem End-To-End With Robust Differentiable Geometric Optimization](https://mlsanity.com/paper/2007.14628/) (2020)
- [Real-time Kinematic Ground Truth for the Oxford RobotCar Dataset](https://mlsanity.com/paper/2002.10152/) (2020)
- [Differentiable Divergences Between Time Series](https://mlsanity.com/paper/2010.08354/) (2020)
- [Learning Discriminative Prototypes with Dynamic Time Warping](https://mlsanity.com/paper/2103.09458/) (2021)
- [Temporal Transformer Networks: Joint Learning of Invariant and Discriminative Time Warping](https://mlsanity.com/paper/1906.05947/) (2019)
- [Probabilistic Visual Place Recognition for Hierarchical Localization](https://mlsanity.com/paper/2105.03091/) (2021)
- [Autowarp: Learning a Warping Distance from Unlabeled Time Series Using Sequence Autoencoders](https://mlsanity.com/paper/1810.10107/) (2018)
- [NeuralWarp: Time-Series Similarity with Warping Networks](https://mlsanity.com/paper/1812.08306/) (2018)

## Built on by (most cited first, 5 of 5)

- [Applications of Spiking Neural Networks in Visual Place Recognition](https://mlsanity.com/paper/2311.13186/) (2023)
- [Revisiting Implicit Differentiation for Learning Problems in Optimal Control](https://mlsanity.com/paper/2310.14468/) (2023)
- [LPGD: A General Framework for Backpropagation through Embedded Optimization Layers](https://mlsanity.com/paper/2407.05920/) (2024)
- [TimePoint: Accelerated Time Series Alignment via Self-Supervised Keypoint and Descriptor Learning](https://mlsanity.com/paper/2505.23475/) (2025)
- [Geometry-Conditioned Fixed-Scaffold Encoders for Time-Warp Robust Sequence Retrieval](https://mlsanity.com/paper/2609.36809/) (2026)

## Abstract (arXiv)

This paper addresses learning end-to-end models for time series data that include a temporal alignment step via dynamic time warping (DTW). Existing approaches to differentiable DTW either differentiate through a fixed warping path or apply a differentiable relaxation to the min operator found in the recursive steps used to solve the DTW problem. We instead propose a DTW layer based around bi-level optimisation and deep declarative networks, which we name DecDTW. By formulating DTW as a continuous, inequality constrained optimisation problem, we can compute gradients for the solution of the optimal alignment (with respect to the underlying time series) using implicit differentiation. An interesting byproduct of this formulation is that DecDTW outputs the optimal warping path between two time series as opposed to a soft approximation, recoverable from Soft-DTW. We show that this property is particularly useful for applications where downstream loss functions are defined on the optimal alignment path itself. This naturally occurs, for instance, when learning to improve the accuracy of predicted alignments against ground truth alignments. We evaluate DecDTW on two such applications, namely the audio-to-score alignment task in music information retrieval and the visual place recognition task in robotics, demonstrating state-of-the-art results in both.