SVGraph: Learning Semantic Graphs from Instructional Videos
Not on the map. Why?
Abstract (arXiv)
In this work, we focus on generating graphical representations of noisy, instructional videos for video understanding. We propose a self-supervised, interpretable approach that does not require any annotations for graphical representations, which would be expensive and time consuming to collect. We attempt to overcome "black box" learning limitations by presenting Semantic Video Graph or SVGraph, a multi-modal approach that utilizes narrations for semantic interpretability of the learned graphs. SVGraph 1) relies on the agreement between multiple modalities to learn a unified graphical structure with the help of cross-modal attention and 2) assigns semantic interpretation with the help of Semantic-Assignment, which captures the semantics from video narration. We perform experiments on multiple datasets and demonstrate the interpretability of SVGraph in semantic graph learning.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Builds on 60 of its 77 references that are in the map. First posted on arXiv 16 Jul 2022.
Builds on
60 of 77 references.
References · 60
- 2017Attention Is All You Needcited by 59,932
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
- 2017Neural Message Passing for Quantum Chemistrycited by 3,460
- 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
- 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2016Xception: Deep Learning with Depthwise Separable Convolutionscited by 2,320
- 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
- 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
38 more
- 2018CosFace: Large Margin Cosine Loss for Deep Face Recognitioncited by 968
- 2019Strategies for Pre-training Graph Neural Networkscited by 964
- 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
- 2017In Defense of the Triplet Loss for Person Re-Identificationcited by 784
- 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
- 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
- 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 656
- 2015Cyclical Learning Rates for Training Neural Networkscited by 627
- 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
- 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
- 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
- 2017Scene Graph Generation by Iterative Message Passingcited by 462
- 2017Look, Listen and Learncited by 452
- 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
- 2019COIN: A Large-scale Dataset for Comprehensive Instructional Video Analysiscited by 316
- 2018Graph R-CNN for Scene Graph Generationcited by 296
- 2020Self-Supervised MultiModal Versatile Networkscited by 276
- 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
- 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
- 2017Objects that Soundcited by 254
- 2017Unsupervised Representation Learning by Sorting Sequencescited by 219
- 2020SpeedNet: Learning the Speediness in Videoscited by 171
- 2019Knowledge-Embedded Routing Network for Scene Graph Generationcited by 163
- 2017Scene Graph Generation from Objects, Phrases and Region Captionscited by 154
- 2017Suggestive Annotation: A Deep Active Learning Framework for Biomedical Image Segmentationcited by 145
- 2020Self-supervised Video Representation Learning by Pace Predictioncited by 139
- 2017Deep learning from crowdscited by 105
- 2020Video Playback Rate Perception for Self-supervisedSpatio-Temporal Representation Learningcited by 103
- 2017Detecting Visual Relationships with Deep Relational Networkscited by 97
- 2018Timeception for Complex Action Recognitioncited by 89
- 2022Self-Supervised Learning for Videos: A Surveycited by 85
- 2020Video Representation Learning by Recognizing Temporal Transformationscited by 82
- 2018Factorizable Net: An Efficient Subgraph-based Framework for Scene Graph Generationcited by 78
- 2019Unsupervised learning of action classes with continuous temporal embeddingcited by 75
- 2018Unsupervised Learning and Segmentation of Complex Activities from Videocited by 61
- 2019You Only Watch Once: A Unified CNN Architecture for Real-Time Spatiotemporal Action Localizationcited by 58
- 2019VideoGraph: Recognizing Minutes-Long Human Activities in Videoscited by 57
- 2021Understanding and Mitigating Annotation Bias in Facial Expression Recognitioncited by 50
50 most cited of 60 All 60 (JSON)
Built on by
Citing papers per year: 2023: 1; 2024: 1.
2 papers.