# IA-VLA: Input Augmentation for Vision-Language-Action models in settings with semantically complex tasks

arXiv: https://arxiv.org/abs/2509.24768 · HTML: https://mlsanity.com/paper/2509.24768/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Generalist VLA models and benchmarks, one of 13 directions in Vision-language-action models.
- Builds on 24 of its 28 references that are in the map.
- First posted on arXiv 29 Sep 2025.

## Placed in

- Vision-language-action models / [Generalist VLA models and benchmarks](https://mlsanity.com/vla/generalist-vla-models-and-benchmarks/)

## Builds on (most cited first, 24 of 24)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control](https://mlsanity.com/paper/2307.15818/) (2023)
- [OpenVLA: An Open-Source Vision-Language-Action Model](https://mlsanity.com/paper/2406.09246/) (2024)
- [SAM 2: Segment Anything in Images and Videos](https://mlsanity.com/paper/2408.00714/) (2024)
- [π0: A Vision-Language-Action Flow Model for General Robot Control](https://mlsanity.com/paper/2410.24164/) (2024)
- [RT-1: Robotics Transformer for Real-World Control at Scale](https://mlsanity.com/paper/2212.06817/) (2022)
- [Octo: An Open-Source Generalist Robot Policy](https://mlsanity.com/paper/2405.12213/) (2024)
- [Fine-Tuning Vision-Language-Action Models: Optimizing Speed and Success](https://mlsanity.com/paper/2502.19645/) (2025)
- [A Generalist Agent](https://mlsanity.com/paper/2205.06175/) (2022)
- [BC-Z: Zero-Shot Task Generalization with Robotic Imitation Learning](https://mlsanity.com/paper/2202.02005/) (2022)
- [CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulation](https://mlsanity.com/paper/2411.19650/) (2024)
- [TinyVLA: Towards Fast, Data-Efficient Vision-Language-Action Models for Robotic Manipulation](https://mlsanity.com/paper/2409.12514/) (2024)
- [Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4V](https://mlsanity.com/paper/2310.11441/) (2023)
- [Open-World Object Manipulation using Pre-trained Vision-Language Models](https://mlsanity.com/paper/2303.00905/) (2023)
- [OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction](https://mlsanity.com/paper/2503.03734/) (2025)
- [RoboGround: Robotic Manipulation with Grounded Vision-Language Priors](https://mlsanity.com/paper/2504.21530/) (2025)
- [Instant Policy: In-Context Imitation Learning via Graph Diffusion](https://mlsanity.com/paper/2411.12633/) (2024)
- [Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robust](https://mlsanity.com/paper/2410.01971/) (2024)
- [RoboNurse-VLA: Robotic Scrub Nurse System based on Vision-Language-Action Model](https://mlsanity.com/paper/2409.19590/) (2024)
- [Transferring Foundation Models for Generalizable Robotic Manipulation](https://mlsanity.com/paper/2306.05716/) (2023)
- [Object-Centric Instruction Augmentation for Robotic Manipulation](https://mlsanity.com/paper/2401.02814/) (2024)
- [Language-Guided Object-Centric Diffusion Policy for Generalizable and Collision-Aware Robotic Manipulation](https://mlsanity.com/paper/2407.00451/) (2024)
- [Learning Efficient and Robust Language-conditioned Manipulation using Textual-Visual Relevancy and Equivariant Language Mapping](https://mlsanity.com/paper/2406.15677/) (2024)
- [ROSO: Improving Robotic Policy Inference via Synthetic Observations](https://mlsanity.com/paper/2311.16680/) (2023)

## Built on by (most cited first, 4 of 4)

- [CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decoding](https://mlsanity.com/paper/2601.02295/) (2026)
- [StructSAM: Structure- and Spectrum-Preserving Token Merging for Segment Anything Models](https://mlsanity.com/paper/2603.07307/) (2026)
- [Closing the Lab-to-Store Gap: A Data-Efficient Post-Training and Experience-Driven Learning VLA Framework for Retail Humanoids](https://mlsanity.com/paper/2607.20345/) (2026)
- [DeicticVLA: Unifying Instruction Modes Based on Language and Deictic Gestures in a Single VLA](https://mlsanity.com/paper/2608.28108/) (2026)

## Abstract (arXiv)

Vision-language-action models (VLAs) have become an increasingly popular approach for addressing robot manipulation problems in recent years. However, such models need to output actions at a rate suitable for robot control, which limits the size of the language model they can be based on, and consequently, their language understanding capabilities. Manipulation tasks may require complex language instructions, such as identifying target objects by their relative positions, to specify human intention. Therefore, we introduce IA-VLA, a framework that utilizes the extensive language understanding of a large vision language model as a pre-processing stage to generate improved context to augment the input of a VLA. We evaluate the framework on a set of semantically complex tasks which have been underexplored in VLA literature, namely tasks involving visual duplicates, i.e., visually indistinguishable objects. A dataset of three types of scenes with duplicate objects is used to compare a baseline VLA against two augmented variants. The experiments show that the VLA benefits from the augmentation scheme, especially when faced with language instructions that require the VLA to extrapolate from concepts it has seen in the demonstrations. For the code, dataset, and videos, see https://sites.google.com/view/ia-vla.