mlsanityData to 7 Oct 2026

TubeDETR: Spatio-Temporal Video Grounding with Transformers

arXiv 2203.16434 ↗First posted 30 Mar 2022cs.CV cs.CL cs.LGcited by 97

Not on the map. Why?

Abstract (arXiv)

We consider the problem of localizing a spatio-temporal tube in a video corresponding to a given text query. This is a challenging task that requires the joint and efficient modeling of temporal, spatial and multi-modal interactions. To address this task, we propose TubeDETR, a transformer-based architecture inspired by the recent success of such models for text-conditioned object detection. Our model notably includes: (i) an efficient video and text encoder that models spatial multi-modal interactions over sparsely sampled frames and (ii) a space-time decoder that jointly performs spatio-temporal localization. We demonstrate the advantage of our proposed components through an extensive ablation study. We also evaluate our full approach on the spatio-temporal video grounding task and demonstrate improvements over the state of the art on the challenging VidSTG and HC-STVG benchmarks. Code and trained models are publicly available at https://antoyang.github.io/tubedetr.html.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). Builds on 91 of its 106 references that are in the map. First posted on arXiv 30 Mar 2022.

Builds on

91 of 106 references.

References · 91

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  8. 2020End-to-End Object Detection with Transformerscited by 7,384
  9. 2016Layer Normalizationcited by 5,579
  10. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  11. 2020Longformer: The Long-Document Transformercited by 3,362
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
38 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2018SlowFast Networks for Video Recognitioncited by 1,862
  3. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  4. 2020Reformer: The Efficient Transformercited by 1,658
  5. 2021ViViT: A Video Vision Transformercited by 1,610
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  7. 2020Rethinking Attention with Performerscited by 1,584
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  10. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  11. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  12. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  13. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  14. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,175
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 811
  17. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  18. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 696
  19. 2017Localizing Moments in Video with Natural Languagecited by 660
  20. 2017TALL: Temporal Activity Localization via Language Querycited by 581
  21. 2019End-to-End Learning of Visual Representations from Uncurated Instructional Videoscited by 506
  22. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  23. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  24. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  25. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 427
  26. 2021VATT: Transformers for Multimodal Self-Supervised Learning from Raw Video, Audio and Textcited by 427
  27. 2020Multi-modal Transformer for Video Retrievalcited by 395
  28. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  29. 2019Meshed-Memory Transformer for Image Captioningcited by 343
  30. 2020VirTex: Learning Visual Representations from Textual Annotationscited by 338
  31. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  32. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  33. 2019Learning 2D Temporal Adjacent Networks for Moment Localization with Natural Languagecited by 283
  34. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  35. 2021TransVG: End-to-End Visual Grounding with Transformerscited by 269
  36. 2021History Aware Multimodal Transformer for Vision-and-Language Navigationcited by 268
  37. 2020ActBERT: Learning Global-Local Video-Text Representationscited by 267
  38. 2016Modeling Context Between Objects for Referring Expression Understandingcited by 250

50 most cited of 91 All 91 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 10; 2023: 31; 2024: 10; 2025: 27; 2026: 19.

97 papers.

Sort:

By area:

  1. 2023InternVid: A Large-scale Video-Text Dataset for Multimodal Understanding and Generationcited by 488
  2. 2022Multimodal Learning with Transformers: A Surveycited by 399
  3. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  4. 2023MeViS: A Large-scale Benchmark for Video Segmentation with Motion Expressionscited by 225
  5. 2023Query-Dependent Video Representation for Moment Retrieval and Highlight Detectioncited by 166
  6. 2022Language Conditioned Spatial Relation Reasoning for 3D Object Groundingcited by 121
  7. 2023Referring Multi-Object Trackingcited by 91
  8. 2024Toward Visual Grounding: A Surveycited by 52
  9. 2024Context-Guided Spatio-Temporal Video Groundingcited by 45
  10. 2023You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videoscited by 41
  11. 2022Embracing Consistency: A One-Stage Approach for Spatio-Temporal Video Groundingcited by 40
  12. 2023Memory-and-Anticipation Transformer for Online Action Understandingcited by 39
38 more
  1. 2023VidChapters-7M: Video Chapters at Scalecited by 39
  2. 2023Gazeformer: Scalable, Effective and Fast Prediction of Goal-Directed Human Attentioncited by 37
  3. 2025MeViS: A Multi-Modal Dataset for Referring Motion Expression Video Segmentationcited by 34
  4. 2023Video-GroundingDINO: Towards Open-Vocabulary Spatio-Temporal Video Groundingcited by 32
  5. 2022Learning to Answer Visual Questions from Web Videoscited by 30
  6. 2025SpaceVLLM: Endowing Multimodal Large Language Model with Spatio-Temporal Video Grounding Capabilitycited by 28
  7. 2022CONE: An Efficient COarse-to-fiNE Alignment Framework for Long Video Temporal Groundingcited by 27
  8. 2023Towards Video Anomaly Retrieval from Video Anomaly Detection: New Benchmarks and Modelcited by 25
  9. 2023Type-to-Track: Retrieve Any Object via Prompt-based Trackingcited by 24
  10. 2023Text-Conditioned Resampler For Long Form Video Understandingcited by 24
  11. 2022Vision+X: A Survey on Multimodal Learning in the Light of Datacited by 22
  12. 2023HIG: Hierarchical Interlacement Graph Approach to Scene Graph Generation in Video Understandingcited by 21
  13. 2023Single-Stage Visual Query Localization in Egocentric Videoscited by 20
  14. 2025Knowing Your Target: Target-Aware Transformer Makes Better Spatio-Temporal Video Groundingcited by 17
  15. 2023Meta-Personalizing Vision-Language Models to Find Named Instances in Videocited by 16
  16. 2024Text-controlled Motion Mamba: Text-Instructed Temporal Grounding of Human Motioncited by 16
  17. 2023Dense Object Grounding in 3D Scenescited by 15
  18. 2022Grounded Video Situation Recognitioncited by 13
  19. 2023Look, Remember and Reason: Grounded reasoning in videos with language modelscited by 13
  20. 2024M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehensioncited by 11
  21. 2025STPro: Spatial and Temporal Progressive Learning for Weakly Supervised Spatio-Temporal Groundingcited by 11
  22. 2023What, when, and where? -- Self-Supervised Spatio-Temporal Grounding in Untrimmed Multi-Action Videos from Narrated Instructionscited by 10
  23. 2023Dense Video Object Captioning from Disjoint Supervisioncited by 10
  24. 2025Contextual Self-paced Learning for Weakly Supervised Spatio-Temporal Video Groundingcited by 10
  25. 2025Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Groundingcited by 9
  26. 2025VideoMiner: Iteratively Grounding Key Frames of Hour-Long Videos via Tree-based Group Relative Policy Optimizationcited by 9
  27. 2023D3G: Exploring Gaussian Prior for Temporal Sentence Grounding with Glance Annotationcited by 8
  28. 2023LLM4VG: Large Language Models Evaluation for Video Groundingcited by 8
  29. 2025PRVQL: Progressive Knowledge-guided Refinement for Robust Egocentric Visual Query Localizationcited by 8
  30. 2025TOGA: Temporally Grounded Open-Ended Video QA with Weak Supervisioncited by 8
  31. 2022Towards Parameter-Efficient Integration of Pre-Trained Language Models In Temporal Video Groundingcited by 7
  32. 2025OmniSTVG: Toward Spatio-Temporal Omni-Object Video Groundingcited by 7
  33. 2023MINOTAUR: Multi-task Video Grounding From Multimodal Queriescited by 6
  34. 2025Talk2Event: Grounded Understanding of Dynamic Scenes from Event Camerascited by 6
  35. 2025ToG-Bench: Task-Oriented Spatio-Temporal Grounding in Egocentric Videoscited by 6
  36. 2023REACT: Recognize Every Action Everywhere All At Oncecited by 5
  37. 2025Large-scale Pre-training for Grounded Video Caption Generationcited by 5
  38. 2022Video Referring Expression Comprehension via Transformer with Content-aware Querycited by 4

50 most cited of 97 All 97 (JSON)