# Weakly-Supervised Alignment of Video With Text

arXiv: https://arxiv.org/abs/1505.06027 · HTML: https://mlsanity.com/paper/1505.06027/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 May 2015.

## Built on by (most cited first, 21 of 21)

- [Temporal Action Segmentation: An Analysis of Modern Techniques](https://mlsanity.com/paper/2210.10352/) (2022)
- [Temporal Alignment Networks for Long-term Video](https://mlsanity.com/paper/2204.02968/) (2022)
- [TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrieval](https://mlsanity.com/paper/2104.08271/) (2021)
- [Temporal Query Networks for Fine-grained Video Understanding](https://mlsanity.com/paper/2104.09496/) (2021)
- [CaptainCook4D: A Dataset for Understanding Errors in Procedural Activities](https://mlsanity.com/paper/2312.14556/) (2023)
- [Learning Procedure-aware Video Representation from Instructional Videos and Their Narrations](https://mlsanity.com/paper/2303.17839/) (2023)
- [Look for the Change: Learning Object States and State-Modifying Actions from Untrimmed Web Videos](https://mlsanity.com/paper/2203.11637/) (2022)
- [Unsupervised Temporal Video Grounding with Deep Semantic Clustering](https://mlsanity.com/paper/2201.05307/) (2022)
- [Learning to Ground Instructional Articles in Videos through Narrations](https://mlsanity.com/paper/2306.03802/) (2023)
- [Aligning Subtitles in Sign Language Videos](https://mlsanity.com/paper/2105.02877/) (2021)
- [A Survey on Integrated Sensing, Communication, and Computation](https://mlsanity.com/paper/2408.08074/) (2024)
- [OTAS: Unsupervised Boundary Detection for Object-Centric Temporal Action Segmentation](https://mlsanity.com/paper/2309.06276/) (2023)
- [A Comprehensive Review of the Video-to-Text Problem](https://mlsanity.com/paper/2103.14785/) (2021)
- [Deep Visual Forced Alignment: Learning to Align Transcription with Talking Face Video](https://mlsanity.com/paper/2303.08670/) (2023)
- [Procedure Learning via Regularized Gromov-Wasserstein Optimal Transport](https://mlsanity.com/paper/2507.15540/) (2025)
- [ActAlign: Zero-Shot Fine-Grained Video Classification via Language-Guided Sequence Alignment](https://mlsanity.com/paper/2506.22967/) (2025)
- [Video alignment using unsupervised learning of local and global features](https://mlsanity.com/paper/2304.06841/) (2023)
- [Conditional Gradients for the Approximate Vanishing Ideal](https://mlsanity.com/paper/2202.03349/) (2022)
- [Hierarchical Action Learning for Weakly-Supervised Action Segmentation](https://mlsanity.com/paper/2602.24275/) (2026)
- [From Routes to Steps: Separating Semantic Progress from Local Execution in Vision-and-Language Navigation](https://mlsanity.com/paper/2608.03143/) (2026)
- [A Stitch in Time: Learning Procedural Workflow via Self-Supervised Plackett-Luce Ranking](https://mlsanity.com/paper/2511.17805/) (2025)

## Abstract (arXiv)

Suppose that we are given a set of videos, along with natural language descriptions in the form of multiple sentences (e.g., manual annotations, movie scripts, sport summaries etc.), and that these sentences appear in the same temporal order as their visual counterparts. We propose in this paper a method for aligning the two modalities, i.e., automatically providing a time stamp for every sentence. Given vectorial features for both video and text, we propose to cast this task as a temporal assignment problem, with an implicit linear mapping between the two feature modalities. We formulate this problem as an integer quadratic program, and solve its continuous convex relaxation using an efficient conditional gradient algorithm. Several rounding procedures are proposed to construct the final integer solution. After demonstrating significant improvements over the state of the art on the related task of aligning video with symbolic labels [7], we evaluate our method on a challenging dataset of videos with associated textual descriptions [36], using both bag-of-words and continuous representations for text.