# Mitigating Coordinate Prediction Bias from Positional Encoding Failures

arXiv: https://arxiv.org/abs/2510.22102 · HTML: https://mlsanity.com/paper/2510.22102/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 41 of its 51 references that are in the map.
- First posted on arXiv 25 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 30 of 41)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Classifier-Free Diffusion Guidance](https://mlsanity.com/paper/2207.12598/) (2022)
- [RoFormer: Enhanced Transformer with Rotary Position Embedding](https://mlsanity.com/paper/2104.09864/) (2021)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Qwen Technical Report](https://mlsanity.com/paper/2309.16609/) (2023)
- [LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day](https://mlsanity.com/paper/2306.00890/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Unbiased Scene Graph Generation from Biased Training](https://mlsanity.com/paper/2002.11949/) (2020)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Rotary Position Embedding for Vision Transformer](https://mlsanity.com/paper/2403.13298/) (2024)
- [ShowUI: One Vision-Language-Action Model for GUI Visual Agent](https://mlsanity.com/paper/2411.17465/) (2024)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [LLaVA-Plus: Learning to Use Tools for Creating Multimodal Agents](https://mlsanity.com/paper/2311.05437/) (2023)
- [Multimodal Web Navigation with Instruction-Finetuned Foundation Models](https://mlsanity.com/paper/2305.11854/) (2023)
- [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024)
- [Found in the Middle: Calibrating Positional Attention Bias Improves Long Context Utilization](https://mlsanity.com/paper/2406.16008/) (2024)
- [Long-Context Language Modeling with Parallel Context Encoding](https://mlsanity.com/paper/2402.16617/) (2024)
- [Beyond LLaVA-HD: Diving into High-Resolution Large Multimodal Models](https://mlsanity.com/paper/2406.08487/) (2024)
- [Eliminating Position Bias of Language Models: A Mechanistic Approach](https://mlsanity.com/paper/2407.01100/) (2024)
- [Fourier Position Embedding: Enhancing Attention's Periodic Extension for Length Generalization](https://mlsanity.com/paper/2412.17739/) (2024)

## Built on by (most cited first, 2 of 2)

- [LenGuard-GPC: Length Guarding with Guided-Prompt Consistency for Spatial Reasoning Reinforce Learning](https://mlsanity.com/paper/2607.17243/) (2026)
- [MHVF: Training-Free GUI Grounding via Verification-Guided Multi-Hypothesis Search](https://mlsanity.com/paper/2610.17792/) (2026)

## Abstract (arXiv)

While Multimodal Large Language Models (MLLMs) excel at general vision-language tasks, precise coordinate prediction remains a significant challenge, particularly as high-resolution inputs cause visual positional encodings (VPEs) to degrade. We demonstrate that these encoding failures do not result in random noise but instead trigger predictable, directional biases, suggesting that models default to internal spatial priors when grounding signals are weak. To counteract this, we introduce Vision-PE Shuffle Guidance (VPSG), a training-free, inference-time correction method. VPSG isolates position-unconditioned tendencies by shuffling VPEs and utilizes this negative evidence to steer digit decoding through a lightweight finite-state machine. Evaluation on the ScreenSpot-Pro benchmark confirms that VPSG effectively rectifies coordinate drift, yielding consistent improvements in localization accuracy across various model scales without any retraining. Our code is available at https://github.com/taoxj2001/VPSG.