mlsanityData to 7 Oct 2026

COOT: Cooperative Hierarchical Transformer for Video-Text Representation Learning

arXiv 2011.00597 ↗First posted 1 Nov 2020cs.CV cs.AI cs.CLcited by 100

Not on the map. Why?

Abstract (arXiv)

Many real-world video-text tasks involve different levels of granularity, such as frames and words, clip and sentences or videos and paragraphs, each with distinct semantics. In this paper, we propose a Cooperative hierarchical Transformer (COOT) to leverage this hierarchy information and model the interactions between different levels of granularity and different modalities. The method consists of three major components: an attention-aware feature aggregation layer, which leverages the local temporal context (intra-level, e.g., within a clip), a contextual transformer to learn the interactions between low-level and high-level semantics (inter-level, e.g. clip-video, sentence-paragraph), and a cross-modal cycle-consistency loss to connect video and text. The resulting method compares favorably to the state of the art on several benchmarks while having few parameters. All code is available open-source at https://github.com/gingsi/coot-videotext

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). First posted on arXiv 1 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 24; 2022: 35; 2023: 24; 2024: 10; 2025: 5; 2026: 2.

100 papers.

Sort:

By area:

  1. 2021Transformers in Vision: A Surveycited by 1,009
  2. 2021VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understandingcited by 596
  3. 2022Multimodal Learning with Transformers: A Surveycited by 399
  4. 2022TM2T: Stochastic and Tokenized Modeling for the Reciprocal Generation of 3D Human Motions and Textscited by 357
  5. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  6. 2021VLM: Task-agnostic Video-Language Model Pre-training for Video Understandingcited by 102
  7. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 97
  8. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  9. 2022Self-Supervised Learning for Videos: A Surveycited by 85
  10. 2022Bridging Video-text Retrieval with Multiple Choice Questionscited by 79
  11. 2021CrossCLR: Cross-modal Contrastive Learning For Multi-modal Video Representationscited by 72
  12. 2023HierVL: Learning Hierarchical Video-Language Embeddingscited by 66
38 more
  1. 2022Video Transformers: A Surveycited by 63
  2. 2022Long-Form Video-Language Pre-Training with Multimodal Temporal Contrastive Learningcited by 59
  3. 2024TR-DETR: Task-Reciprocal Transformer for Joint Moment Retrieval and Highlight Detectioncited by 59
  4. 2023Self-Supervised Multimodal Learning: A Surveycited by 58
  5. 2023MomentDiff: Generative Video Moment Retrieval from Random to Realcited by 58
  6. 2025GAM-Agent: Game-Theoretic and Uncertainty-Aware Collaboration for Complex Visual Reasoningcited by 56
  7. 2023Test of Time: Instilling Video-Language Models with a Sense of Timecited by 51
  8. 2023Instruct-ReID: A Multi-purpose Person Re-identification Task with Instructionscited by 46
  9. 2022LocVTP: Video-Text Pre-training for Temporal Localizationcited by 44
  10. 2021Co-training Transformer with Videos and Images Improves Action Recognitioncited by 41
  11. 2022Clover: Towards A Unified Video-Language Alignment and Fusion Modelcited by 41
  12. 2022Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Groundingcited by 40
  13. 2022Hierarchical Local-Global Transformer for Temporal Sentence Groundingcited by 38
  14. 2023Video-Mined Task Graphs for Keystep Recognition in Instructional Videoscited by 35
  15. 2022Reading-strategy Inspired Visual Representation Learning for Text-to-Video Retrievalcited by 34
  16. 2022Improving Cross-Modal Retrieval with Set of Diverse Embeddingscited by 33
  17. 2021HANet: Hierarchical Alignment Networks for Video-Text Retrievalcited by 31
  18. 2022MILES: Visual BERT Pre-training with Injected Language Semantics for Video-text Retrievalcited by 29
  19. 2023Learning to Ground Instructional Articles in Videos through Narrationscited by 28
  20. 2023Text with Knowledge Graph Augmented Transformer for Video Captioningcited by 27
  21. 2022VLTinT: Visual-Linguistic Transformer-in-Transformer for Coherent Video Paragraph Captioningcited by 25
  22. 2022Video-Text Representation Learning via Differentiable Weak Temporal Alignmentcited by 24
  23. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  24. 2021Masking Modalities for Cross-modal Video Retrievalcited by 23
  25. 2021V2C: Visual Voice Cloningcited by 23
  26. 2022Robustness Analysis of Video-Language Models Against Visual and Language Perturbationscited by 23
  27. 2023Opening the Vocabulary of Egocentric Actionscited by 23
  28. 2023Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignmentcited by 21
  29. 2022Integrating Language Guidance into Vision-based Deep Metric Learningcited by 19
  30. 2022VLCap: Vision-Language with Contrastive Learning for Coherent Video Paragraph Captioningcited by 19
  31. 2023Models See Hallucinations: Evaluating the Factuality in Video Captioningcited by 19
  32. 2022Localized Vision-Language Matching for Open-vocabulary Object Detectioncited by 17
  33. 2021Dyadformer: A Multi-modal Transformer for Long-Range Modeling of Dyadic Interactionscited by 16
  34. 2021AI Ethics Statements -- Analysis and lessons learnt from NeurIPS Broader Impact Statementscited by 16
  35. 2023Vision-Language Models can Identify Distracted Driver Behavior from Naturalistic Videoscited by 16
  36. 2022TempCLR: Temporal Alignment Representation with Contrastive Learningcited by 15
  37. 2023Deep Learning for Video-Text Retrieval: a Reviewcited by 15
  38. 2023MELTR: Meta Loss Transformer for Learning to Fine-tune Video Foundation Modelscited by 15

50 most cited of 100 All 100 (JSON)