mlsanityData to 7 Oct 2026

Object-aware Video-language Pre-training for Retrieval

arXiv 2112.00656 ↗First posted 1 Dec 2021cs.CV cs.CLcited by 57

Not on the map. Why?

Abstract (arXiv)

Recently, by introducing large-scale dataset and strong transformer network, video-language pre-training has shown great success especially for retrieval. Yet, existing video-language transformer models do not explicitly fine-grained semantic align. In this work, we present Object-aware Transformers, an object-centric approach that extends video-language transformer to incorporate object representations. The key idea is to leverage the bounding boxes and object tags to guide the training process. We evaluate our model on three standard sub-tasks of video-text matching on four widely used benchmarks. We also provide deep analysis and detailed ablation about the proposed method. We show clear improvement in performance across all tasks and datasets considered, demonstrating the value of a model that incorporates object representations into a video-language architecture. The code will be released at urlhttps://github.com/FingerRec/OA-Transformer.

Key facts. Cited by 57 papers in the map (data through 7 Oct 2026). Builds on 37 of its 46 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

37 of 46 references.

References · 37

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  6. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
25 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  3. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  4. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 946
  5. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  6. 2017Localizing Moments in Video with Natural Languagecited by 660
  7. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 645
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2018Stacked Cross Attention for Image-Text Matchingcited by 414
  10. 2020Multi-modal Transformer for Video Retrievalcited by 395
  11. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  12. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  13. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  14. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 258
  15. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  16. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  17. 2015A Dataset for Movie Descriptioncited by 195
  18. 2016Movie Descriptioncited by 180
  19. 2020Support-set bottlenecks for video-text representation learningcited by 170
  20. 2018Learning a Text-Video Embedding from Incomplete and Heterogeneous Datacited by 103
  21. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  22. 2014Translating Videos to Natural Language Using Deep Recurrent Neural Networkscited by 95
  23. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  24. 2020Noise Estimation Using Density Estimation for Self-Supervised Multimodal Learningcited by 74
  25. 2020Video Understanding as Machine Translationcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 23; 2023: 18; 2024: 9; 2025: 3; 2026: 3.

57 papers.

Sort:

By area:

  1. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  2. 2022Egocentric Video-Language Pretrainingcited by 231
  3. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  4. 2022All in One: Exploring Unified Video-Language Pre-trainingcited by 171
  5. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 138
  6. 2023EgoVLPv2: Egocentric Video-Language Pre-training with Fusion in the Backbonecited by 130
  7. 2023Foundational Models Defining a New Era in Vision: A Survey and Outlookcited by 109
  8. 2022VindLU: A Recipe for Effective Video-and-Language Pretrainingcited by 87
  9. 2022Masked Image Modeling with Denoising Contrastcited by 45
  10. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  11. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  12. 2022ECLIPSE: Efficient Long-range Video Retrieval using Sight and Soundcited by 39
38 more
  1. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  2. 2022VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrievalcited by 36
  3. 2023Affordance Grounding from Demonstration Video to Target Imagecited by 34
  4. 2022AssistQ: Affordance-centric Question-driven Task Completion for Egocentric Assistantcited by 32
  5. 2024Enhancing Video-Language Representations with Structural Spatio-Temporal Alignmentcited by 32
  6. 2023A Large Cross-Modal Video Retrieval Dataset with Reading Comprehensioncited by 31
  7. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  8. 2022Position-guided Text Prompt for Vision-Language Pre-trainingcited by 27
  9. 2022SimVTP: Simple Video Text Pre-training with Masked Autoencoderscited by 21
  10. 2021Video-Text Pre-training with Learned Regionscited by 20
  11. 2024Chrono: A Simple Blueprint for Representing Time in MLLMscited by 17
  12. 2022Towards Fast Adaptation of Pretrained Contrastive Models for Multi-channel Video-Language Retrievalcited by 16
  13. 2022TempCLR: Temporal Alignment Representation with Contrastive Learningcited by 15
  14. 2023Deep Learning for Video-Text Retrieval: a Reviewcited by 15
  15. 2022RoME: Role-aware Mixture-of-Expert Transformer for Text-to-Video Retrievalcited by 11
  16. 2022Pre-training image-language transformers for open-vocabulary taskscited by 11
  17. 2022Masked Contrastive Pre-Training for Efficient Video-Text Retrievalcited by 11
  18. 2023COSA: Concatenated Sample Pretrained Vision-Language Foundation Modelcited by 11
  19. 2023Gaussian Mixture Proposals with Pull-Push Learning Scheme to Capture Diverse Events for Weakly Supervised Temporal Video Groundingcited by 10
  20. 2022LaT: Latent Translation with Cycle-Consistency for Video-Text Retrievalcited by 9
  21. 2024EA-VTR: Event-Aware Video-Text Retrievalcited by 8
  22. 2023Learning Trajectory-Word Alignments for Video-Language Taskscited by 7
  23. 2023Video-Text Retrieval by Supervised Sparse Multi-Grained Learningcited by 7
  24. 2023MultiVENT: Multilingual Videos of Events with Aligned Natural Textcited by 7
  25. 2024TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigmcited by 6
  26. 2025EvdCLIP: Improving Vision-Language Retrieval with Entity Visual Descriptions from Large Language Modelscited by 6
  27. 2022RaP: Redundancy-aware Video-language Pre-training for Text-Video Retrievalcited by 5
  28. 2023STOA-VLP: Spatial-Temporal Modeling of Object and Action for Video-Language Pre-trainingcited by 5
  29. 2023EtC: Temporal Boundary Expand then Clarify for Weakly Supervised Video Grounding with Multimodal Large Language Modelcited by 4
  30. 2022MuMUR : Multilingual Multimodal Universal Retrievalcited by 3
  31. 2022Efficient Cross-Modal Video Retrieval with Meta-Optimized Framescited by 2
  32. 2023LANDMARK: Language-guided Representation Enhancement Framework for Scene Graph Generationcited by 2
  33. 2024Unified Video-Language Pre-training with Synchronized Audiocited by 2
  34. 2022Unified Loss of Pair Similarity Optimization for Vision-Language Retrievalcited by 1
  35. 2024SNP-S3: Shared Network Pre-training and Significant Semantic Strengthening for Various Video-Text Taskscited by 1
  36. 2024Unifying Latent and Lexicon Representations for Effective Video-Text Retrievalcited by 1
  37. 2024Grounding Descriptions in Images informs Zero-Shot Visual Recognitioncited by 1
  38. 2025Leveraging Modality Tags for Enhanced Cross-Modal Video Retrievalcited by 1

50 most cited of 57 All 57 (JSON)