mlsanityData to 7 Oct 2026

TRIPS: Efficient Vision-and-Language Pre-training with Text-Relevant Image Patch Selection

arXiv 2305.04474 ↗First posted 8 May 2023cs.CV cs.AIcited by 18

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) have been widely used in large-scale Vision and Language Pre-training (VLP) models. Though previous VLP works have proved the effectiveness of ViTs, they still suffer from computational efficiency brought by the long visual sequence. To tackle this problem, in this paper, we propose an efficient vision-and-language pre-training model with Text-Relevant Image Patch Selection, namely TRIPS, which reduces the visual sequence progressively with a text-guided patch-selection layer in the visual backbone for efficient training and inference. The patch-selection layer can dynamically compute text-dependent visual attention to identify the attentive image tokens with text guidance and fuse inattentive ones in an end-to-end manner. Meanwhile, TRIPS does not introduce extra parameters to ViTs. Experimental results on a variety of popular benchmark datasets demonstrate that TRIPS gain a speedup of 40% over previous similar VLP models, yet with competitive or better downstream task performance.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 43 of its 52 references that are in the map. First posted on arXiv 8 May 2023.

Builds on

43 of 52 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  7. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  8. 2017Mask R-CNNcited by 7,259
  9. 2015You Only Look Once: Unified, Real-Time Object Detectioncited by 4,829
  10. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  12. 2015VQA: Visual Question Answeringcited by 2,489
31 more
  1. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  2. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  3. 2020Reformer: The Efficient Transformercited by 1,658
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  6. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  7. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,224
  10. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  11. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  12. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  13. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  14. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 849
  15. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 646
  16. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  17. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  18. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  19. 2021Rethinking Spatial Dimensions of Vision Transformerscited by 361
  20. 2022Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizationscited by 359
  21. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  22. 2019UNITER: UNiversal Image-TExt Representation Learningcited by 225
  23. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  24. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 185
  25. 2021Evo-ViT: Slow-Fast Token Evolution for Dynamic Vision Transformercited by 175
  26. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 109
  27. 2021Compressing Visual-linguistic Model via Knowledge Distillationcited by 85
  28. 2021E2E-VLP: End-to-End Vision-Language Pre-training Enhanced by Visual Learningcited by 84
  29. 2020MiniVLM: A Smaller and Faster Vision-Language Modelcited by 51
  30. 2021Playing Lottery Tickets with Vision and Languagecited by 45
  31. 2021Self-slimmed Vision Transformercited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 10; 2024: 4; 2025: 3; 2026: 1.

18 papers.

Sort:

By area:

  1. 2023Hallucination Augmented Contrastive Learning for Multimodal Large Language Modelcited by 119
  2. 2023X-InstructBLIP: A Framework for aligning X-Modal instruction-aware representations to LLMs and Emergent Cross-modal Reasoningcited by 78
  3. 2023CrossGET: Cross-Guided Ensemble of Tokens for Accelerating Vision-Language Transformerscited by 44
  4. 2024Accelerating Transformers with Spectrum-Preserving Token Mergingcited by 35
  5. 2025Shifting AI Efficiency From Model-Centric to Data-Centric Compressioncited by 29
  6. 2024Hal-Eval: A Universal and Fine-grained Hallucination Evaluation Framework for Large Vision Language Modelscited by 27
  7. 2024Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Large Modelscited by 13
  8. 2023Turbo: Informativity-Driven Acceleration Plug-In for Vision-Language Modelscited by 11
  9. 2023Exploiting Pseudo Image Captions for Multimodal Summarizationcited by 10
  10. 2023SmartTrim: Adaptive Tokens and Attention Pruning for Efficient Vision-Language Modelscited by 7
  11. 2023BUS:Efficient and Effective Vision-language Pre-training with Bottom-Up Patch Summarizationcited by 5
  12. 2023COPA: Efficient Vision-Language Pre-training Through Collaborative Object- and Patch-Text Alignmentcited by 5
6 more
  1. 2023TiMix: Text-aware Image Mixing for Effective Vision-Language Pre-trainingcited by 3
  2. 2025Guiding Cross-Modal Representations with MLLM Priors via Preference Alignmentcited by 3
  3. 2023ELIP: Efficient Discriminative Language-Image Pre-training with Fewer Vision Tokenscited by 2
  4. 2024Improving the Robustness of Knowledge-Grounded Dialogue via Contrastive Learningcited by 2
  5. 2025PCRI: Measuring Context Robustness in Multimodal Models for Enterprise Applicationscited by 1
  6. 2026MedARC: Training-Free Adaptive Redundancy Compression of Visual Tokens for 3D Medical Vision-Language Modelscited by 0