mlsanityData to 14 Oct 2026

Circle-RoPE: Cone-like Decoupled Rotary Positional Embedding for Large Vision-Language Models

arXiv 2505.16416 ↗First posted 22 May 2025cs.CV cs.AIcited by 14

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Rotary Position Embedding (RoPE) is widely adopted in large language models, but when applied to vision-language models (VLMs) it couples text and image position indices and can introduce spurious cross-modal relative-position bias. We propose Per-Token Distance (PTD) to quantify cross-modal positional disentanglement, and prove that PTD = 0 is a sufficient condition to eliminate the geometric attention bias induced by RoPE. Guided by this criterion, we introduce Circle-RoPE, which remaps 2D image-token coordinates onto an annulus orthogonal to the text position axis, yielding a cone-like geometry where each text token is equidistant to all image tokens while preserving intra-image spatial structure. We further propose Alternating Geometry Encoding (AGE) to combine complementary geometric priors by alternating the decoupled geometry of Circle-RoPE and the grid-based prior of standard RoPE across layers. This design enables cross-modal positional disentanglement while preserving fine-grained intra-image spatial structure. Experiments on diverse VLM backbones and multimodal benchmarks show consistent gains in spatial grounding and visual reasoning. The code is available at https://github.com/lose4578/CircleRoPE.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 32 of its 40 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

32 of 40 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025On Path to Multimodal Generalist: General-Level and General-Bench7 May
  2. 2025Pixel-SAIL: Single Transformer For Pixel-Grounded Understanding14 Apr
  3. 2025The Scalability of Simplicity: Empirical Analysis of Vision-Language Learning with a Single Transformer14 Apr
  4. 2025Eve: Efficient Multimodal Vision Language Models with Elastic Visual Experts8 Jan
  5. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding13 Dec
  6. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scaling6 Dec
  7. 2024MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale6 Dec
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution18 Sep
  9. 2024MiniCPM-V: A GPT-4V Level MLLM on Your Phone3 Aug
  10. 2024Ovis: Structural Embedding Alignment for Multimodal Large Language Model31 May
  11. 2023LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents9 Nov
  12. 2023MMBench: Is Your Multi-modal Model an All-around Player?12 Jul

Other references · 20

  1. 2023Visual Instruction Tuningcited by 7,869
  2. 2025Qwen2.5-VL Technical Reportcited by 5,587
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  5. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  6. 2019Towards VQA Models That Can Readcited by 1,718
  7. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,614
  8. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
  9. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 914
  10. 2016A Diagram Is Worth A Dozen Imagescited by 804
  11. 2024Emu3: Next-Token Prediction is All You Needcited by 781
  12. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
8 more
  1. 2024VLMEvalKit: An Open-Source Toolkit for Evaluating Large Multi-Modality Modelscited by 455
  2. 2024mPLUG-Owl3: Towards Long Image-Sequence Understanding in Multi-Modal Large Language Modelscited by 291
  3. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  4. 2023Transformer-Based Visual Segmentation: A Surveycited by 124
  5. 2025Baichuan-Omni-1.5 Technical Reportcited by 111
  6. 2025VideoRoPE: What Makes for Good Video Rotary Position Embedding?cited by 65
  7. 2025Scalable Vision Language Model Training via High Quality Data Curationcited by 55
  8. 2023DiffNAS: Bootstrapping Diffusion Models by Prompting for Better Architecturescited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 11.

14 papers.

Sort:

By area:

  1. 2025Revisiting Multimodal Positional Encoding in Vision-Language Modelscited by 16
  2. 2026VLA-ATTC: Adaptive Test-Time Compute for VLA Models with Relative Action Critic Modelcited by 9
  3. 2026Spiral RoPE: Rotate Your Rotary Positional Embeddings in the 2D Planecited by 7
  4. 2025Improving GUI Grounding with Explicit Position-to-Coordinate Mappingcited by 4
  5. 2026Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recoverycited by 4
  6. 2026RAP: KV-Cache Compression via RoPE-Aligned Pruningcited by 3
  7. 2026SoPE: Spherical Coordinate-Based Positional Embedding for Enhancing Spatial Perception of 3D LVLMscited by 2
  8. 2026MODIX: A Training-Free Multimodal Information-Driven Positional Index Scaling for Vision-Language Modelscited by 2
  9. 2025AttAnchor: Guiding Cross-Modal Token Alignment in VLMs with Attention Anchorscited by 1
  10. 2026Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encodingcited by 1
  11. 2026DPPE: Rethinking Camera-Based Positional Encoding for Scaling Multi-View Transformerscited by 1
  12. 2026Tinted Frames: Question Framing Blinds Vision-Language Modelscited by 0
2 more
  1. 2026Mitigating Mask Prior Drift and Positional Attention Collapse in Large Diffusion Vision-Language Modelscited by 0
  2. 2026RIG-RoPE: Relation-Stratified Multimodal Attention with Instance-Local Rotary Geometry and Representation-Aware Traversal Coordinatescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HoPE: Hybrid of Position Embedding for Long Context Vision-Language Models
  2. 2025VRoPE: Rotary Position Embedding for Video Large Language Models
  3. 2026C^2ROPE: Causal Continuous Rotary Positional Encoding for 3D Large Multimodal-Models Reasoning
  4. 2026Layer-Aware Position Embeddings for Visual Token Pruning in Multimodal Large Language Models
  5. 2025ID-Align: RoPE-Conscious Position Remapping for Dynamic High-Resolution Adaptation in Vision-Language Models
  6. 2026IVC-Prune: Revealing the Implicit Visual Coordinates in LVLMs for Vision Token Pruning
  7. 2025OMEGA: Optimized Multimodal Position Encoding Index Derivation with Global Adaptive Scaling for Vision-Language Models
  8. 2025Towards Long-window Anchoring in Vision-Language Model Distillation