# Chain-of-Procedure: Hierarchical Visual-Language Reasoning for Procedural QA

arXiv: https://arxiv.org/abs/2605.14928 · HTML: https://mlsanity.com/paper/2605.14928/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 21 of its 52 references that are in the map.
- First posted on arXiv 14 May 2026.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 21 of 21)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyond](https://mlsanity.com/paper/2308.12966/) (2023)
- [LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models](https://mlsanity.com/paper/2403.13372/) (2024)
- [Gemini: A Family of Highly Capable Multimodal Models](https://mlsanity.com/paper/2312.11805/) (2023)
- [Tracking State Changes in Procedural Text: A Challenge Dataset and Models for Process Paragraph Comprehension](https://mlsanity.com/paper/1805.06975/) (2018)
- [Visual Goal-Step Inference using wikiHow](https://mlsanity.com/paper/2104.05845/) (2021)
- [M-Longdoc: A Benchmark For Multimodal Super-Long Document Understanding And A Retrieval-Aware Tuning Framework](https://mlsanity.com/paper/2411.06176/) (2024)
- [CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation](https://mlsanity.com/paper/2406.10462/) (2024)
- [MG-LLaVA: Towards Multi-Granularity Visual Instruction Tuning](https://mlsanity.com/paper/2406.17770/) (2024)
- [VisionGPT: Vision-Language Understanding Agent Using Generalized Multimodal Framework](https://mlsanity.com/paper/2403.09027/) (2024)
- [SnapNTell: Enhancing Entity-Centric Visual Question Answering with Retrieval Augmented Multimodal LLM](https://mlsanity.com/paper/2403.04735/) (2024)
- [A Recipe for Creating Multimodal Aligned Datasets for Sequential Tasks](https://mlsanity.com/paper/2005.09606/) (2020)
- [AI Governance and Accountability: An Analysis of Anthropic's Claude](https://mlsanity.com/paper/2407.01557/) (2024)
- [LLaVA-Chef: A Multi-modal Generative Model for Food Recipes](https://mlsanity.com/paper/2408.16889/) (2024)
- [Not All LoRA Parameters Are Essential: Insights on Inference Necessity](https://mlsanity.com/paper/2503.23360/) (2025)
- [OWLViz: An Open-World Benchmark for Visual Question Answering](https://mlsanity.com/paper/2503.07631/) (2025)

## Built on by (most cited first, 2 of 2)

- [Seeing is not Enough: Vision-Language Models Perceive Evidence but Fail to Act](https://mlsanity.com/paper/2609.22588/) (2026)
- [Tasks over Application Manuals: Revealing Gaps in Long-Horizon Procedural Reasoning for Language Models](https://mlsanity.com/paper/2609.13005/) (2026)

## Abstract (arXiv)

Recent advances in vision-language models (VLMs) have achieved impressive results on standard image-text tasks, yet their potential for visual procedure question answering (VP-QA) remains largely unexplored. VP-QA presents unique challenges where users query next-step actions by uploading images for intermediate states of complex procedures. To systematically evaluate VLMs on this practical task, we propose ProcedureVQA, a novel multimodal benchmark specifically designed for visual procedural reasoning. Through comprehensive analysis, we identify two critical limitations in current VLMs: inadequate cross-modal retrieval of structured procedures given visual states, and misalignment between image sequence granularity and textual step decomposition. To address these issues, we present Chain-of-Procedure (CoP), a hierarchical reasoning framework that first retrieves relevant instructions using visual cues, then performs step refinement through semantic decomposition, and finally generates the next step. Experiments across six VLMs demonstrate CoP's effectiveness, achieving up to 13% absolute improvement over standard baselines.