mlsanityData to 14 Oct 2026

TS2-Net: Token Shift and Selection Transformer for Text-Video Retrieval

arXiv 2207.07852 ↗First posted 16 Jul 2022cs.CVcited by 104

Not on the map. Why?

Abstract (arXiv)

Text-Video retrieval is a task of great practical value and has received increasing attention, among which learning spatial-temporal video representation is one of the research hotspots. The video encoders in the state-of-the-art video retrieval models usually directly adopt the pre-trained vision backbones with the network structure fixed, they therefore can not be further improved to produce the fine-grained spatial-temporal video representation. In this paper, we propose Token Shift and Selection Network (TS2-Net), a novel token shift and selection transformer architecture, which dynamically adjusts the token sequence and selects informative tokens in both temporal and spatial dimensions from input video samples. The token shift module temporally shifts the whole token features back-and-forth across adjacent frames, to preserve the complete token representation and capture subtle movements. Then the token selection module selects tokens that contribute most to local spatial semantics. Based on thorough experiments, the proposed TS2-Net achieves state-of-the-art performance on major text-video retrieval benchmarks, including new records on MSRVTT, VATEX, LSMDC, ActivityNet, and DiDeMo.

Key facts. Cited by 104 papers in the map (data through 14 Oct 2026). Builds on 46 of its 53 references that are in the map. First posted on arXiv 16 Jul 2022.

Builds on

46 of 53 references.

References · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  4. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,270
  5. 2018SlowFast Networks for Video Recognitioncited by 1,867
  6. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,620
  7. 2021ViViT: A Video Vision Transformercited by 1,620
  8. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,492
  9. 2021Frozen in Time: A Joint Video and Image Encoder for End-to-End Retrievalcited by 1,177
  10. 2021Video Swin Transformercited by 1,091
  11. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  12. 2019HowTo100M: Learning a Text-Video Embedding by Watching Hundred Million Narrated Video Clipscited by 950
34 more
  1. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  2. 2018TSM: Temporal Shift Module for Efficient Video Understandingcited by 856
  3. 2017Dense-Captioning Events in Videoscited by 793
  4. 2017Localizing Moments in Video with Natural Languagecited by 663
  5. 2021CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrievalcited by 651
  6. 2017Rethinking Spatiotemporal Feature Learning: Speed-Accuracy Trade-offs in Video Classificationcited by 521
  7. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  8. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  9. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  10. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 414
  11. 2020Multi-modal Transformer for Video Retrievalcited by 395
  12. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  13. 2020UniVL: A Unified Video and Language Pre-Training Model for Multimodal Understanding and Generationcited by 259
  14. 2021CLIP2Video: Mastering Video-Text Retrieval via Image CLIPcited by 225
  15. 2019Use What You Have: Video Retrieval Using Representations From Collaborative Expertscited by 214
  16. 2018A Joint Sequence Fusion Model for Video Question Answering and Retrievalcited by 208
  17. 2016Movie Descriptioncited by 181
  18. 2020Fine-grained Video-Text Retrieval with Hierarchical Graph Reasoningcited by 180
  19. 2020Support-set bottlenecks for video-text representation learningcited by 170
  20. 2021Improving Video-Text Retrieval by Multi-Stream Corpus Alignment and Dual Softmax Losscited by 116
  21. 2018Cross-Modal and Hierarchical Modeling of Video and Textcited by 103
  22. 2021T2VLAD: Global-Local Sequence Alignment for Text-Video Retrievalcited by 103
  23. 2021TACo: Token-aware Cascade Contrastive Learning for Video-Text Alignmentcited by 102
  24. 2017Offline bilingual word vectors, orthogonal transformations and the inverted softmaxcited by 96
  25. 2021HiT: Hierarchical Transformer with Momentum Contrast for Video-Text Retrievalcited by 87
  26. 2018Dual Encoding for Zero-Example Video Retrievalcited by 85
  27. 2021Space-time Mixing Attention for Video Transformercited by 84
  28. 2021TEACHTEXT: CrossModal Generalized Distillation for Text-Video Retrievalcited by 81
  29. 2021MDMMT: Multidomain Multimodal Transformer for Video Retrievalcited by 78
  30. 2021Cross Modal Retrieval with Querybank Normalisationcited by 72
  31. 2021Differentiable Patch Selection for Image Recognitioncited by 68
  32. 2021Token Shift Transformer for Video Classificationcited by 61
  33. 2021Efficient Video Transformers with Spatial-Temporal Token Selectioncited by 59
  34. 2019A Strong and Robust Baseline for Text-Image Matchingcited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 35; 2024: 26; 2025: 28; 2026: 6.

104 papers.

Sort:

By area:

  1. 2022InternVideo: General Video Foundation Models via Generative and Discriminative Learningcited by 447
  2. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 203
  3. 2023VALOR: Vision-Audio-Language Omni-Perception Pretraining Model and Datasetcited by 140
  4. 2022Cap4Video: What Can Auxiliary Captions Do for Text-Video Retrieval?cited by 90
  5. 2025M-LLM Based Video Frame Selection for Efficient Video Understandingcited by 88
  6. 2022CLIP-Driven Fine-grained Text-Image Person Re-identificationcited by 79
  7. 2024Koala: Key frame-conditioned long video-LLMcited by 57
  8. 2023UATVR: Uncertainty-Adaptive Text-Video Retrievalcited by 55
  9. 2023DiffusionRet: Generative Text-Video Retrieval with Diffusion Modelcited by 52
  10. 2023CoVR-2: Automatic Data Construction for Composed Video Retrievalcited by 52
  11. 2025Token Reduction Should Go Beyond Efficiency in Generative Models -- From Vision, Language to Multimodalitycited by 50
  12. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
38 more
  1. 2023Video-Text as Game Players: Hierarchical Banzhaf Interaction for Cross-Modal Representation Learningcited by 45
  2. 2023CLIP-guided Prototype Modulating for Few-shot Action Recognitioncited by 41
  3. 2024TempMe: Video Temporal Token Merging for Efficient Text-Video Retrievalcited by 40
  4. 2024On Learning Multi-Modal Forgery Representation for Diffusion Generated Video Detectioncited by 40
  5. 2022VoP: Text-Video Co-operative Prompt Tuning for Cross-Modal Retrievalcited by 36
  6. 2022Cross-Modal Adapter for Vision-Language Retrievalcited by 35
  7. 2023Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrievalcited by 35
  8. 2024Magic Tokens: Select Diverse Tokens for Multi-modal Object Re-Identificationcited by 35
  9. 2024Text Is MASS: Modeling as Stochastic Embedding for Text-Video Retrievalcited by 33
  10. 2023Text-Video Retrieval with Disentangled Conceptualization and Set-to-Set Alignmentcited by 32
  11. 2023DreamInpainter: Text-Guided Subject-Driven Image Inpainting with Diffusion Modelscited by 31
  12. 2024DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrievalcited by 26
  13. 2024Composed Video Retrieval via Enriched Context and Discriminative Embeddingscited by 26
  14. 2023ViLA: Efficient Video-Language Alignment for Video Question Answeringcited by 23
  15. 2022SimVTP: Simple Video Text Pre-training with Masked Autoencoderscited by 21
  16. 2023Audio-Enhanced Text-to-Video Retrieval using Text-Conditioned Feature Alignmentcited by 21
  17. 2023Balance Act: Mitigating Hubness in Cross-Modal Retrieval with Query and Gallery Bankscited by 20
  18. 2025Video-ColBERT: Contextualized Late Interaction for Text-to-Video Retrievalcited by 20
  19. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  20. 2024RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adaptercited by 16
  21. 2023RGNet: A Unified Clip Retrieval and Grounding Network for Long Videoscited by 15
  22. 2024Streaming Video Diffusion: Online Video Editing with Diffusion Modelscited by 13
  23. 2024Reversed in Time: A Novel Temporal-Emphasized Benchmark for Cross-Modal Video-Text Retrievalcited by 13
  24. 2024Towards Efficient and Effective Text-to-Video Retrieval with Coarse-to-Fine Visual Representation Learningcited by 12
  25. 2025Continual Text-to-Video Retrieval with Frame Fusion and Task-Aware Routingcited by 12
  26. 2023Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learningcited by 11
  27. 2023Dual-Modal Attention-Enhanced Text-Video Retrieval with Triplet Partial Margin Contrastive Learningcited by 10
  28. 2023RTQ: Rethinking Video-language Understanding Based on Image-text Modelcited by 8
  29. 2025Learning Audio-guided Video Representation with Gated Attention for Video-Text Retrievalcited by 8
  30. 2022PRVR: Partially Relevant Video Retrievalcited by 7
  31. 2023Learning Trajectory-Word Alignments for Video-Language Taskscited by 7
  32. 2023Video-Text Retrieval by Supervised Sparse Multi-Grained Learningcited by 7
  33. 2023An Empirical Study of Frame Selection for Text-to-Video Retrievalcited by 6
  34. 2024SHE-Net: Syntax-Hierarchy-Enhanced Text-Video Retrievalcited by 6
  35. 2024Learning text-to-video retrieval from image captioningcited by 6
  36. 2024TokenBinder: Text-Video Retrieval with One-to-Many Alignment Paradigmcited by 6
  37. 2025NeighborRetr: Balancing Hub Centrality in Cross-Modal Retrievalcited by 6
  38. 2023Mask to reconstruct: Cooperative Semantics Completion for Video-text Retrievalcited by 5

50 most cited of 104 All 104 (JSON)