mlsanityData to 14 Oct 2026

PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models

arXiv 2205.11169 ↗First posted 23 May 2022cs.CV cs.AI cs.CLcited by 30

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language pre-training (VLP) has shown impressive performance on a wide range of cross-modal tasks, where VLP models without reliance on object detectors are becoming the mainstream due to their superior computation efficiency and competitive performance. However, the removal of object detectors also deprives the capability of VLP models in explicit object modeling, which is essential to various position-sensitive vision-language (VL) tasks, such as referring expression comprehension and visual commonsense reasoning. To address the challenge, we introduce PEVL that enhances the pre-training and prompt tuning of VLP models with explicit object position modeling. Specifically, PEVL reformulates discretized object positions and language in a unified language modeling framework, which facilitates explicit VL alignment during pre-training, and also enables flexible prompt tuning for various downstream tasks. We show that PEVL enables state-of-the-art performance of detector-free VLP models on position-sensitive tasks such as referring expression comprehension and phrase grounding, and also improves the performance on position-insensitive tasks with grounded inputs. We make the data and code for this paper publicly available at https://github.com/thunlp/PEVL.

Key facts. Cited by 30 papers in the map (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 61 of its 95 references that are in the map. First posted on arXiv 23 May 2022.

Builds on

61 of 95 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modeling23 Nov
  2. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Concepts16 Nov
  3. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervision24 Aug

Other references · 58

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  7. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  8. 2016Grad-CAM: Visual Explanations from Deep Networks via Gradient-based Localizationcited by 6,733
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
  12. 2015VQA: Visual Question Answeringcited by 2,506
38 more
  1. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  2. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
  3. 2019Language Models as Knowledge Bases?cited by 1,936
  4. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  5. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  6. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,362
  7. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  8. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  9. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  10. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  11. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  12. 2016Modeling Context in Referring Expressionscited by 1,063
  13. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  14. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  15. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  16. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  17. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  18. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  19. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 661
  20. 2018From Recognition to Cognition: Visual Commonsense Reasoningcited by 556
  21. 2021DenseCLIP: Language-Guided Dense Prediction with Context-Aware Promptingcited by 532
  22. 2017Scene Graph Generation by Iterative Message Passingcited by 462
  23. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  24. 2018MAttNet: Modular Attention Network for Referring Expression Comprehensioncited by 431
  25. 2017Neural Motifs: Scene Graph Parsing with Global Contextcited by 414
  26. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  27. 2021Learning How to Ask: Querying LMs with Mixtures of Soft Promptscited by 377
  28. 2020Unbiased Scene Graph Generation from Biased Trainingcited by 356
  29. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 349
  30. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  31. 2021ActionCLIP: A New Paradigm for Video Action Recognitioncited by 320
  32. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  33. 2021Prompting Visual-Language Models for Efficient Video Understandingcited by 310
  34. 2018Graph R-CNN for Scene Graph Generationcited by 298
  35. 2018Bilinear Attention Networkscited by 280
  36. 2018Learning to Compose Dynamic Tree Structures for Visual Contextscited by 259
  37. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  38. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 3; 2023: 21; 2024: 3; 2025: 2; 2026: 1.

30 papers.

Sort:

By area:

  1. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  2. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  3. 2023GPT4RoI: Instruction Tuning Large Language Model on Region-of-Interestcited by 302
  4. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  5. 2023NExT-Chat: An LMM for Chat, Detection and Segmentationcited by 80
  6. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Modelscited by 59
  7. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  8. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  9. 2023Panoptic Scene Graph Generation with Semantics-Prototype Learningcited by 42
  10. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  11. 2023Meta-Adapter: An Online Few-shot Learner for Vision-Language Modelcited by 31
  12. 2023Lenna: Language Enhanced Reasoning Detection Assistantcited by 29
18 more
  1. 2023Voila-A: Aligning Vision-Language Models with User's Gaze Attentioncited by 28
  2. 2023RegionBLIP: A Unified Multi-modal Pre-training Framework for Holistic and Regional Comprehensioncited by 27
  3. 2022Prompt Tuning for Generative Multimodal Pretrained Modelscited by 25
  4. 2023Measuring Progress in Fine-grained Vision-and-Language Understandingcited by 25
  5. 2023Pre-training on Synthetic Driving Data for Trajectory Predictioncited by 18
  6. 2025Visual Position Prompt for MLLM based Visual Groundingcited by 16
  7. 2023Unified Visual Relationship Detection with Vision and Language Modelscited by 13
  8. 2023Domain-wise Invariant Learning for Panoptic Scene Graph Generationcited by 7
  9. 2023Weakly-Supervised Learning of Visual Relations in Multimodal Pretrainingcited by 6
  10. 2023Learning to Agree on Vision Attention for Visual Commonsense Reasoningcited by 2
  11. 2024EventLens: Leveraging Event-Aware Pretraining and Cross-modal Linking Enhances Visual Commonsense Reasoningcited by 2
  12. 2025Pose-Aware Weakly-Supervised Action Segmentationcited by 2
  13. 2023ViM: Vision Middleware for Unified Downstream Transferringcited by 1
  14. 2023TreePrompt: Learning to Compose Tree Prompts for Explainable Visual Groundingcited by 1
  15. 2024ComAlign: Compositional Alignment in Vision-Language Modelscited by 1
  16. 2023SCoRD: Subject-Conditional Relation Detection with Text-Augmented Datacited by 0
  17. 2023Augment the Pairs: Semantics-Preserving Image-Caption Pair Augmentation for Grounding-Based Vision and Language Modelscited by 0
  18. 2026Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Position-guided Text Prompt for Vision-Language Pre-training
  2. 2022VL-CheckList: Evaluating Pre-trained Vision-Language Models with Objects, Attributes and Relations
  3. 2022X2-VLM: All-In-One Pre-trained Model For Vision-Language Tasks
  4. 2022EfficientVLM: Fast and Accurate Vision-Language Models via Knowledge Distillation and Modal-adaptive Pruning
  5. 2022VLP: A Survey on Vision-Language Pre-training
  6. 2023ReForm-Eval: Evaluating Large Vision Language Models via Unified Re-Formulation of Task-Oriented Benchmarks
  7. 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
  8. 2023Stop Pre-Training: Adapt Visual-Language Models to Unseen Languages