# PixelVLA: Advancing Pixel-level Understanding in Vision-Language-Action Model

arXiv: https://arxiv.org/abs/2511.01571 · HTML: https://mlsanity.com/paper/2511.01571/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 55 of its 68 references that are in the map.
- First posted on arXiv 3 Nov 2025.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 30 of 55)

- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Segment Anything](https://mlsanity.com/paper/2304.02643/) (2023)
- [DINOv2: Learning Robust Visual Features without Supervision](https://mlsanity.com/paper/2304.07193/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection](https://mlsanity.com/paper/2303.05499/) (2023)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [Diffusion Policy: Visuomotor Policy Learning via Action Diffusion](https://mlsanity.com/paper/2303.04137/) (2023)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware](https://mlsanity.com/paper/2304.13705/) (2023)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning](https://mlsanity.com/paper/2306.03310/) (2023)
- [Shikra: Unleashing Multimodal LLM's Referential Dialogue Magic](https://mlsanity.com/paper/2306.15195/) (2023)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [Open X-Embodiment: Robotic Learning Datasets and RT-X Models](https://mlsanity.com/paper/2310.08864/) (2023)
- [DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset](https://mlsanity.com/paper/2403.12945/) (2024)
- [RLBench: The Robot Learning Benchmark & Learning Environment](https://mlsanity.com/paper/1909.12271/) (2019)
- [BridgeData V2: A Dataset for Robot Learning at Scale](https://mlsanity.com/paper/2308.12952/) (2023)
- [SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Model](https://mlsanity.com/paper/2501.15830/) (2025)
- [Ferret: Refer and Ground Anything Anywhere at Any Granularity](https://mlsanity.com/paper/2310.07704/) (2023)
- [Evaluating Real-World Robot Manipulation Policies in Simulation](https://mlsanity.com/paper/2405.05941/) (2024)
- [GLaMM: Pixel Grounding Large Multimodal Model](https://mlsanity.com/paper/2311.03356/) (2023)
- [CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models](https://mlsanity.com/paper/2503.22020/) (2025)
- [Robotic Control via Embodied Chain-of-Thought Reasoning](https://mlsanity.com/paper/2407.08693/) (2024)
- [Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models](https://mlsanity.com/paper/2402.07865/) (2024)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [TraceVLA: Visual Trace Prompting Enhances Spatial-Temporal Awareness for Generalist Robotic Policies](https://mlsanity.com/paper/2412.10345/) (2024)

## Built on by (most cited first, 18 of 18)

- [SimVLA: A Simple VLA Baseline for Robotic Manipulation](https://mlsanity.com/paper/2602.18224/) (2026)
- [OmniGuide: Universal Guidance Fields for Enhancing Generalist Robot Policies](https://mlsanity.com/paper/2603.10052/) (2026)
- [Action-Sketcher: From Reasoning to Action via Visual Sketches for Long-Horizon Robotic Manipulation](https://mlsanity.com/paper/2601.01618/) (2026)
- [PlanViz: Evaluating Planning-Oriented Image Generation and Editing for Computer-Use Tasks](https://mlsanity.com/paper/2602.06663/) (2026)
- [Lifelong Language-Conditioned Robotic Manipulation Learning](https://mlsanity.com/paper/2603.05160/) (2026)
- [Lifelong Embodied Navigation Learning](https://mlsanity.com/paper/2603.06073/) (2026)
- [LabVLA: Grounding Vision-Language-Action Models in Scientific Laboratories](https://mlsanity.com/paper/2606.13578/) (2026)
- [Unleashing More Actions via Action Compositional Training for VLA Models](https://mlsanity.com/paper/2607.00351/) (2026)
- [All-day Multi-scenes Lifelong Vision-and-Language Navigation with Tucker Adaptation](https://mlsanity.com/paper/2603.14276/) (2026)
- [AnySlot: Goal-Conditioned Vision-Language-Action Policies for Zero-Shot Slot-Level Placement](https://mlsanity.com/paper/2604.10432/) (2026)
- [MotionVLA: Injecting Geometric Motion into Vision-Language-Action Model](https://mlsanity.com/paper/2606.08288/) (2026)
- [Towards Human-like Physical Intelligence: Lifelong Vision-Language-Action Learning for Robotic Manipulation](https://mlsanity.com/paper/2607.14852/) (2026)
- [SPARC: Reliable Spatial Annotations from Robot Demonstrations at Scale](https://mlsanity.com/paper/2606.13497/) (2026)
- [ResAgent: Entropy-based Prior Point Discovery and Visual Reasoning for Referring Expression Segmentation](https://mlsanity.com/paper/2601.16394/) (2026)
- [FocalPolicy: Frequency-Optimized Chunking and Locally Anchored Flow Matching for Coherent Visuomotor Policy](https://mlsanity.com/paper/2605.15944/) (2026)
- [Metric-Bench: Exploring In-context Spatial Metric Reasoning in VLMs for Indoor Scenes](https://mlsanity.com/paper/2609.25841/) (2026)
- [Triggering Generalist Reasoning via Predictive Uncertainty for Dual-System VLA](https://mlsanity.com/paper/2610.05025/) (2026)
- [RAIN: Robotic Task Generalization with Region-Aware Interaction Networks](https://mlsanity.com/paper/2610.14039/) (2026)

## Abstract (arXiv)

Vision-Language-Action models (VLAs) are emerging as powerful tools for learning generalizable visuomotor control policies. However, current VLAs are mostly trained on large-scale image-text-action data and remain limited in two key ways: (i) they struggle with pixel-level scene understanding, and (ii) they rely heavily on textual prompts, which reduces their flexibility in real-world settings. To address these challenges, we introduce PixelVLA, the first VLA model designed to support both pixel-level reasoning and multimodal prompting with text and visual inputs. Our approach is built on a new visuomotor instruction tuning framework that integrates a multiscale pixel-aware encoder with a visual promptaware encoder. To train PixelVLA effectively, we further propose a two-stage automated annotation pipeline that generates Pixel-160K, a large-scale dataset with pixel-level annotations derived from existing robot data. Experiments on three standard VLA benchmarks and two VLA model variants show that PixelVLA improves manipulation success rates by 10.1%-28.7% over OpenVLA, while requiring only 1.5% of its pretraining cost. These results demonstrate that PixelVLA can be integrated into existing VLAs to enable more accurate, efficient, and versatile robot control in complex environments.