Reconstructing and grounding narrated instructional videos in 3D
Not on the map. Why?
Abstract (arXiv)
Narrated instructional videos often show and describe manipulations of similar objects, e.g., repairing a particular model of a car or laptop. In this work we aim to reconstruct such objects and to localize associated narrations in 3D. Contrary to the standard scenario of instance-level 3D reconstruction, where identical objects or scenes are present in all views, objects in different instructional videos may have large appearance variations given varying conditions and versions of the same product. Narrations may also have large variation in natural language expressions. We address these challenges by three contributions. First, we propose an approach for correspondence estimation combining learnt local features and dense flow. Second, we design a two-step divide and conquer reconstruction approach where the initial 3D reconstructions of individual videos are combined into a 3D alignment graph. Finally, we propose an unsupervised approach to ground natural language in obtained 3D reconstructions. We demonstrate the effectiveness of our approach for the domain of car maintenance. Given raw instructional videos and no manual supervision, our method successfully reconstructs engines of different car models and associates textual descriptions with corresponding objects in 3D.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 57 references that are in the map. First posted on arXiv 9 Sep 2021.
Builds on
29 of 57 references.
References · 29
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
- 2015NetVLAD: CNN architecture for weakly supervised place recognitioncited by 935
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
- 2020Understanding Human Hands in Contact at Internet Scalecited by 281
- 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
- 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
- 2019R2D2: Repeatable and Reliable Detector and Descriptorcited by 206
- 2019Cross-task weakly supervised learning from instructional videoscited by 199
- 2018Neighbourhood Consensus Networkscited by 173
- 2015Learning Deep Structure-Preserving Image-Text Embeddingscited by 139
17 more
- 2019GLU-Net: Global-Local Universal Network for Dense Flow and Correspondencescited by 139
- 2017Learning Two-Branch Neural Networks for Image-Text Matching Taskscited by 119
- 2015Unsupervised Learning from Narrated Instruction Videoscited by 116
- 2014Deep Fragment Embeddings for Bidirectional Image Sentence Mappingcited by 99
- 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
- 2016Connectionist Temporal Modeling for Weakly Supervised Action Labelingcited by 70
- 2017Weakly Supervised Action Learning with RNN based Fine-to-coarse Modelingcited by 70
- 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
- 2017End-to-end weakly-supervised semantic alignmentcited by 60
- 2019Hyperpixel Flow: Semantic Correspondence with Multi-layer Neural Featurescited by 56
- 2017Conditional Image-Text Embedding Networkscited by 50
- 2017FCSS: Fully Convolutional Self-Similarity for Dense Semantic Correspondencecited by 42
- 2017Weakly-supervised Visual Grounding of Phrases with Linguistic Structurescited by 40
- 2015What's Cookin'? Interpreting Cooking Videos using Text, Speech and Visioncited by 31
- 2017Joint Discovery of Object States and Manipulation Actionscited by 28
- 2018Finding beans in burgers: Deep semantic-visual embedding with localizationcited by 28
- 2017Unsupervised Visual-Linguistic Reference Resolution in Instructional Videoscited by 4
Built on by
Not cited yet.