mlsanityData to 7 Oct 2026

NAVERO: Unlocking Fine-Grained Semantics for Video-Language Compositionality

arXiv 2408.09511 ↗First posted 18 Aug 2024cs.CVcited by 1

Not on the map. Why?

Abstract (arXiv)

We study the capability of Video-Language (VidL) models in understanding compositions between objects, attributes, actions and their relations. Composition understanding becomes particularly challenging for video data since the compositional relations rapidly change over time in videos. We first build a benchmark named AARO to evaluate composition understanding related to actions on top of spatial concepts. The benchmark is constructed by generating negative texts with incorrect action descriptions for a given video and the model is expected to pair a positive text with its corresponding video. Furthermore, we propose a training method called NAVERO which utilizes video-text data augmented with negative texts to enhance composition understanding. We also develop a negative-augmented visual-language matching loss which is used explicitly to benefit from the generated negative text. We compare NAVERO with other state-of-the-art methods in terms of compositional understanding as well as video-text retrieval performance. NAVERO achieves significant improvement over other methods for both video-language and image-language composition understanding, while maintaining strong performance on traditional text-video retrieval tasks.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 47 of its 59 references that are in the map. First posted on arXiv 18 Aug 2024.

Builds on

47 of 59 references.

References · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
  7. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  8. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  9. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  10. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  11. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  12. 2018Relational inductive biases, deep learning, and graph networkscited by 1,341
35 more
  1. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2017Localizing Moments in Video with Natural Languagecited by 660
  4. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  5. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  6. 2022When and why vision-language models behave like bags-of-words, and what to do about it?cited by 603
  7. 2018End-to-End Incremental Learningcited by 596
  8. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 523
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  11. 2017Measuring Catastrophic Forgetting in Neural Networkscited by 359
  12. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  13. 2018Videos as Space-Time Region Graphscited by 221
  14. 2021VIOLET : End-to-End Video-Language Transformers with Masked Visual-token Modelingcited by 187
  15. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  16. 2021Align and Prompt: Video-and-Language Pre-training with Entity Promptscited by 149
  17. 2022OmniVL:One Foundation Model for Image-Language and Video-Language Taskscited by 143
  18. 2021VALSE: A Task-Independent Benchmark for Vision and Language Models Centered on Linguistic Phenomenacited by 134
  19. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relationscited by 113
  20. 2021Learning to Predict Visual Attributes in the Wildcited by 95
  21. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  22. 2020Grounded Situation Recognitioncited by 77
  23. 2023Dense and Aligned Captions (DAC) Promote Compositional Reasoning in VL Modelscited by 72
  24. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  25. 2018Object Level Visual Reasoning in Videoscited by 67
  26. 2022Masked Vision and Language Modeling for Multi-modal Representation Learningcited by 66
  27. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  28. 2018Mapping Images to Scene Graphs with Permutation-Invariant Structured Predictioncited by 44
  29. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  30. 2019HAKE: Human Activity Knowledge Enginecited by 32
  31. 2023Coarse-to-Fine Contrastive Learning in Image-Text-Graph Space for Improved Vision-Language Compositionalitycited by 32
  32. 2018Referring Relationshipscited by 29
  33. 2020Dynamic and Static Context-aware LSTM for Multi-agent Motion Predictioncited by 21
  34. 2022LiteVL: Efficient Video-Language Learning with Enhanced Spatial-Temporal Modelingcited by 16
  35. 2023ICSVR: Investigating Compositional and Syntactic Understanding in Video Retrieval Modelscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Zero-Shot Temporal Action Localization Through Textual Guidancecited by 0