A Persian Vision-Language-Action Dataset and Benchmark for Ambiguity- and Infeasibility-Aware Instruction-to-Plan Grounding
On the map: Vision-language-action models → Reasoning and chain-of-thought in VLA also fits: Human-video and latent-action pretraining
Abstract (arXiv)
Converting natural-language instructions into executable action plans conditioned on a scene image is a core capability for embodied human-robot interaction, but existing benchmarks target English, simulated or video-grounded environments, and largely assume instructions are clear and executable. We introduce the first Persian vision-language-action dataset for this task: 258 instances over 230 real household/office photographs, annotated with reference plans, deliberate ambiguity and infeasibility labels, disambiguating question-answer pairs, and an auto-generated chain-of-thought rationale for training. Using this resource, we ablate retrieval-augmentation, chain-of-thought prompting, and an agentic disambiguation step, evaluated with an LLM-as-judge protocol and a new metric, PGOS-FA, extending prior work (PG2S) with step-order sensitivity, an over-generation penalty, and Persian-adapted embeddings. Paired statistical testing across 36 comparisons shows retrieval-augmentation is a large, robustly significant effect, while finer-grained additions are directionally consistent but not statistically distinguishable from noise at this scale, and the LLM-judge metric has near-zero power where PGOS-FA succeeds. A blind human-correlation study on 79 real outputs shows PGOS-FA tracks human judgment at least as closely as the LLM-judge. Testing generalization beyond the ablation's single backbone, we find that most of the gap between an open-weight and a substantially stronger closed planner is attributable to open-weight model generation rather than to the closed model's capability specifically, and that our best configuration significantly outperforms an independently reproduced external baseline despite a weaker backbone. The dataset, PGOS-FA implementation, and evaluation code will be made available upon reasonable request.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reasoning and chain-of-thought in VLA, one of 13 directions in Vision-language-action models. Builds on 21 of its 26 references that are in the map. First posted on arXiv 11 Oct 2026.
Builds on
21 of 26 references.
References · 21
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
- 2021Ego4D: Around the World in 3,000 Hours of Egocentric Videocited by 1,556
- 2024DROID: A Large-Scale In-The-Wild Robot Manipulation Datasetcited by 852
- 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
- 2020Stanza: A Python Natural Language Processing Toolkit for Many Human Languagescited by 723
- 2022LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Modelscited by 462
- 2024Understanding the planning of LLM agents: A surveycited by 421
- 2020Room-Across-Room: Multilingual Vision-and-Language Navigation with Dense Spatiotemporal Groundingcited by 414
- 2023EmbodiedGPT: Vision-Language Pre-Training via Embodied Chain of Thoughtcited by 279
9 more
- 2025Large Language Model Agent: A Survey on Methodology, Applications and Challengescited by 205
- 2025Structured Uncertainty guided Clarification for LLM Agentscited by 41
- 2024P-RAG: Progressive Retrieval Augmented Generation For Planning on Embodied Everyday Taskcited by 15
- 2024Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMscited by 11
- 2025AmbiK: Dataset of Ambiguous Tasks in Kitchen Environmentcited by 9
- 2024Multi-Agent Planning Using Visual Language Modelscited by 7
- 2026Limited Linguistic Diversity in Embodied AI Datasetscited by 6
- 2024PLATO: Planning with LLMs and Affordances for Tool Manipulationcited by 5
- 2026VLN-NF: Feasibility-Aware Vision-and-Language Navigation with False-Premise Instructionscited by 3
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026CoTinyVLA: Chain-of-Thought Distillation for a Sub-Billion-Parameter Vision-Language-Action Model
- 2026Vision-Language-Action in Robotics: A Survey of Datasets, Benchmarks, and Data Engines
- 2026JEPA-WAM: Connecting Generated Visual Instructions to World Action Models through JEPA Latent Representations
- 2026Grounded Semantic Re-Binding for Robust Instruction Generalization in Vision-Language-Action Models
- 2026LangForce: Bayesian Decomposition of Vision Language Action Models via Latent Action Queries
- 2026RoboIRGBench: Benchmarking Implicit Referential Grounding in Vision-Language-Action Models
- 2026The Yes-Man Syndrome: Benchmarking Abstention in Embodied Robotic Agents
- 2026From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data