# Procedural Knowledge Extraction from Industrial Troubleshooting Guides Using Vision Language Models

arXiv: https://arxiv.org/abs/2601.22754 · HTML: https://mlsanity.com/paper/2601.22754/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 9 of its 15 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 9 of 9)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [DocVQA: A Dataset for VQA on Document Images](https://mlsanity.com/paper/2007.00398/) (2020)
- [FlowLearn: Evaluating Large Vision-Language Models on Flowchart Understanding](https://mlsanity.com/paper/2407.05183/) (2024)
- [Learning Household Task Knowledge from WikiHow Descriptions](https://mlsanity.com/paper/1909.06414/) (2019)
- [Reasoning about Procedures with Natural Language Processing: A Tutorial](https://mlsanity.com/paper/2205.07455/) (2022)
- [Human Evaluation of Procedural Knowledge Graph Extraction from Text with Large Language Models](https://mlsanity.com/paper/2412.03589/) (2024)

## Built on by (most cited first, 1 of 1)

- [FlowExtract: Procedural Knowledge Extraction from Maintenance Flowcharts](https://mlsanity.com/paper/2604.06770/) (2026)

## Abstract (arXiv)

Industrial troubleshooting guides encode diagnostic procedures in flowchart-like diagrams where spatial layout and technical language jointly convey meaning. To integrate this knowledge into operator support systems, which assist shop-floor personnel in diagnosing and resolving equipment issues, the information must first be extracted and structured for machine interpretation. However, when performed manually, this extraction is labor-intensive and error-prone. Vision Language Models offer potential to automate this process by jointly interpreting visual and textual meaning, yet their performance on such guides remains underexplored. This paper evaluates two VLMs on extracting structured knowledge, comparing two prompting strategies: standard instruction-guided versus an augmented approach that cues troubleshooting layout patterns. Results reveal model-specific trade-offs between layout sensitivity and semantic robustness, informing practical deployment decisions.