mlsanityData to 7 Oct 2026

Advancing General Multimodal Capability of Vision-language Models with Pyramid-descent Visual Position Encoding

arXiv 2501.10967 ↗First posted 19 Jan 2025cs.CV cs.AI cs.CLcited by 4

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Vision-language Models (VLMs) have shown remarkable capabilities in advancing general artificial intelligence, yet the irrational encoding of visual positions persists in inhibiting the models' comprehensive perception performance across different levels of granularity. In this work, we propose Pyramid-descent Visual Position Encoding (PyPE), a novel approach designed to enhance the perception of visual tokens within VLMs. By assigning visual position indexes from the periphery to the center and expanding the central receptive field incrementally, PyPE addresses the limitations of traditional raster-scan methods and mitigates the long-term decay effects induced by Rotary Position Embedding (RoPE). Our method reduces the relative distance between interrelated visual elements and instruction tokens, promoting a more rational allocation of attention weights and allowing for a multi-granularity perception of visual elements and countering the over-reliance on anchor tokens. Extensive experimental evaluations demonstrate that PyPE consistently improves the general capabilities of VLMs across various sizes. Code is available at https://github.com/SakuraTroyChen/PyPE.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 56 references that are in the map. First posted on arXiv 19 Jan 2025.

Builds on

25 of 56 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  2. 2024LMMs-Eval: Reality Check on the Evaluation of Large Multimodal Models17 Jul
  3. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI24 Apr
  4. 2024TinyLLaVA: A Framework of Small-scale Large Multimodal Models22 Feb
  5. 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities4 Aug
  6. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul

Other references · 19

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
  7. 2018Self-Attention with Relative Position Representationscited by 1,112
  8. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  10. 2024MiniCPM: Unveiling the Potential of Small Language Models with Scalable Training Strategiescited by 694
  11. 2023Textbooks Are All You Need II: phi-1.5 technical reportcited by 559
  12. 2021Conditional Positional Encodings for Vision Transformerscited by 402
7 more
  1. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 235
  2. 2020How Much Position Information Do Convolutional Neural Networks Encode?cited by 171
  3. 2024FiT: Flexible Vision Transformer for Diffusion Modelcited by 84
  4. 2024MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Trainingcited by 50
  5. 2024Mitigating Object Hallucination via Concentric Causal Attentioncited by 48
  6. 2023LLaVA-Interactive: An All-in-One Demo for Image Chat, Segmentation, Generation and Editingcited by 47
  7. 2024VisionLLaMA: A Unified LLaMA Backbone for Vision Taskscited by 28

Built on by

Cited by year

Citing papers per year: 2025: 4.

4 papers.

  1. 2025Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flowcited by 13
  2. 2025ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Modelscited by 2
  3. 2025Advancing Complex Wide-Area Scene Understanding with Hierarchical Coresets Selectioncited by 1
  4. 2025Mitigating Coordinate Prediction Bias from Positional Encoding Failurescited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025Revisiting Multimodal Positional Encoding in Vision-Language Models
  2. 2025OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models
  3. 2024V2PE: Improving Multimodal Long-Context Capability of Vision-Language Models with Variable Visual Position Encoding
  4. 2025Hidden in plain sight: VLMs overlook their visual representations
  5. 2025Towards Long-window Anchoring in Vision-Language Model Distillation
  6. 2023Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models
  7. 2022PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models
  8. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models