When Did It Happen? Duration-informed Temporal Localization of Narrated Actions in Vlogs
Not on the map. Why?
Abstract (arXiv)
We consider the task of temporal human action localization in lifestyle vlogs. We introduce a novel dataset consisting of manual annotations of temporal localization for 13,000 narrated actions in 1,200 video clips. We present an extensive analysis of this data, which allows us to better understand how the language and visual modalities interact throughout the videos. We propose a simple yet effective method to localize the narrated actions based on their expected duration. Through several experiments and analyses, we show that our method brings complementary information with respect to previous methods, and leads to improvements over previous work for the task of temporal action localization.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 43 of its 64 references that are in the map. First posted on arXiv 16 Feb 2022.
Builds on
43 of 64 references.
References · 43
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
- 2017The Kinetics Human Action Video Datasetcited by 1,894
- 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
- 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
- 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
- 2017Dense-Captioning Events in Videoscited by 789
31 more
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
- 2017Localizing Moments in Video with Natural Languagecited by 660
- 2017TALL: Temporal Activity Localization via Language Querycited by 581
- 2016Hollywood in Homes: Crowdsourcing Data Collection for Activity Understandingcited by 535
- 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
- 2014Long-term Recurrent Convolutional Networks for Visual Recognition and Descriptioncited by 483
- 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
- 2017AVA: A Video Dataset of Spatio-temporally Localized Atomic Visual Actionscited by 449
- 2018Stacked Cross Attention for Image-Text Matchingcited by 414
- 2016YouTube-8M: A Large-Scale Video Classification Benchmarkcited by 324
- 2018Video Action Transformer Networkcited by 316
- 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
- 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
- 2015Stacked Attention Networks for Image Question Answeringcited by 273
- 2016Image Captioning with Semantic Attentioncited by 260
- 2018Moments in Time Dataset: one million videos for event understandingcited by 215
- 2019Cross-task weakly supervised learning from instructional videoscited by 199
- 2018To Find Where You Talk: Temporal Sentence Localization in Video with Attention Based Location Regressioncited by 157
- 2019TVQA+: Spatio-Temporal Grounding for Video Question Answeringcited by 153
- 2018Charades-Ego: A Large-Scale Dataset of Paired Third and First Person Videoscited by 127
- 2017Single Shot Temporal Action Detectioncited by 123
- 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
- 2018MAC: Mining Activity Concepts for Language-based Temporal Localizationcited by 84
- 2018Localizing Moments in Video with Temporal Languagecited by 83
- 2019ExCL: Extractive Clip Localization Using Natural Language Descriptionscited by 82
- 2019GQA: A New Dataset for Real-World Visual Reasoning and Compositional Question Answeringcited by 77
- 2014Learning Spatiotemporal Features with 3D Convolutional Networkscited by 43
- 2017From Lifestyle Vlogs to Everyday Interactionscited by 42
- 2019Multimodal Abstractive Summarization for How2 Videoscited by 41
- 2017What Actions are Needed for Understanding Human Actions in Videos?cited by 29
- 2019Identifying Visible Actions in Lifestyle Vlogscited by 6
Built on by
Citing papers per year: 2023: 1.
1 paper.