mlsanityData to 7 Oct 2026

Video-aided Unsupervised Grammar Induction

arXiv 2104.04369 ↗First posted 9 Apr 2021cs.CV cs.CLcited by 17

Not on the map. Why?

Abstract (arXiv)

We investigate video-aided grammar induction, which learns a constituency parser from both unlabeled text and its corresponding video. Existing methods of multi-modal grammar induction focus on learning syntactic grammars from text-image pairs, with promising results showing that the information from static images is useful in induction. However, videos provide even richer information, including not only static objects but also actions and state changes useful for inducing verb phrases. In this paper, we explore rich features (e.g. action, object, scene, audio, face, OCR and speech) from videos, taking the recent Compound PCFG model as the baseline. We further propose a Multi-Modal Compound PCFG model (MMC-PCFG) to effectively aggregate these rich features from different modalities. Our proposed MMC-PCFG is trained end-to-end and outperforms each individual modality and previous state-of-the-art systems on three benchmarks, i.e. DiDeMo, YouCook2 and MSRVTT, confirming the effectiveness of leveraging video information for unsupervised grammar induction.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 37 of its 56 references that are in the map. First posted on arXiv 9 Apr 2021.

Builds on

37 of 56 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2016Densely Connected Convolutional Networkscited by 6,691
  4. 2017Squeeze-and-Excitation Networkscited by 4,763
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  6. 2013Auto-Encoding Variational Bayescited by 3,980
  7. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  8. 2015SSD: Single Shot MultiBox Detectorcited by 3,125
  9. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  10. 2016Identity Mappings in Deep Residual Networkscited by 1,960
  11. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
25 more
  1. 2016CNN Architectures for Large-Scale Audio Classificationcited by 701
  2. 2017Localizing Moments in Video with Natural Languagecited by 660
  3. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  4. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 604
  5. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  6. 2020Multi-modal Transformer for Video Retrievalcited by 395
  7. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  8. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  9. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  10. 2019Large-scale weakly-supervised pre-training for video action recognitioncited by 130
  11. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  12. 2018Ordered Neurons: Integrating Tree Structures into Recurrent Neural Networkscited by 101
  13. 2020MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioningcited by 79
  14. 2019Compound Probabilistic Context-Free Grammars for Grammar Inductioncited by 76
  15. 2018PixelLink: Detecting Scene Text via Instance Segmentationcited by 60
  16. 2019Unsupervised Latent Tree Induction with Deep Inside-Outside Recursive Autoencoderscited by 60
  17. 2017Neural Language Modeling by Jointly Learning Syntax and Lexiconcited by 57
  18. 2020Learning Modality Interaction for Temporal Sentence Localization and Event Captioning in Videoscited by 56
  19. 2019Visually Grounded Neural Syntax Acquisitioncited by 48
  20. 2020Visually Grounded Compound PCFGscited by 34
  21. 2019Exploiting Temporal Relationships in Video Moment Localization with Natural Languagecited by 33
  22. 2019Cross-Domain Generalization of Neural Constituency Parserscited by 17
  23. 2020What is Learned in Visually Grounded Neural Syntax Acquisitioncited by 15
  24. 2018Unsupervised Grammar Induction with Depth-bounded PCFGcited by 8
  25. 2020The End-of-End-to-End: A Video Understanding Pentathlon Challenge (2020)cited by 6

Built on by

Cited by year

Citing papers per year: 2021: 2; 2022: 8; 2023: 3; 2024: 3; 2025: 1.

17 papers.

Sort:

  1. 2021Sequence-to-Sequence Learning with Latent Neural Grammarscited by 30
  2. 2022VALHALLA: Visual Hallucination for Machine Translationcited by 27
  3. 2025Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrievalcited by 20
  4. 2021Neural Bi-Lexicalized PCFG Inductioncited by 18
  5. 2022Unsupervised Slot Schema Induction for Task-oriented Dialogcited by 11
  6. 2022Dynamic Programming in Rank Space: Scaling Structured Inference with Low-Rank HMMs and PCFGscited by 10
  7. 2022Unsupervised Vision-Language Parsing: Seamlessly Bridging Visual Scene Graphs with Language Structures via Dependency Relationshipscited by 9
  8. 2022Unsupervised Discontinuous Constituency Parsing with Mildly Context-Sensitive Grammarscited by 7
  9. 2022Learning a Grammar Inducer from Massive Uncurated Instructional Videoscited by 3
  10. 2023Simple Hardware-Efficient PCFGs with Independent Left and Right Productionscited by 3
  11. 2022Re-evaluating the Need for Multimodal Signals in Unsupervised Grammar Inductioncited by 2
  12. 2023Audio-Visual Neural Syntax Acquisitioncited by 2
5 more
  1. 2024Learning Language Structures through Groundingcited by 2
  2. 2023On the Transferability of Visually Grounded PCFGscited by 1
  3. 2024Improving Unsupervised Constituency Parsing via Maximizing Semantic Informationcited by 1
  4. 2022Unsupervised and Few-shot Parsing from Pretrained Language Modelscited by 0
  5. 2024Grammar Induction from Visual, Speech and Textcited by 0