# FORGE: Fine-grained Multimodal Evaluation for Manufacturing Scenarios

arXiv: https://arxiv.org/abs/2604.07413 · HTML: https://mlsanity.com/paper/2604.07413/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 19 of its 57 references that are in the map.
- First posted on arXiv 8 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Chart understanding and anomaly detection](https://mlsanity.com/mllm/chart-understanding-and-anomaly-detection/)

## Builds on (most cited first, 19 of 19)

- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [PoseCNN: A Convolutional Neural Network for 6D Object Pose Estimation in Cluttered Scenes](https://mlsanity.com/paper/1711.00199/) (2017)
- [UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interaction](https://mlsanity.com/paper/2503.15661/) (2025)
- [Paper2Poster: Towards Multimodal Poster Automation from Scientific Papers](https://mlsanity.com/paper/2505.21497/) (2025)
- [MMAD: A Comprehensive Benchmark for Multimodal Large Language Models in Industrial Anomaly Detection](https://mlsanity.com/paper/2410.09453/) (2024)
- [Grounding Computer Use Agents on Human Demonstrations](https://mlsanity.com/paper/2511.07332/) (2025)
- [Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs](https://mlsanity.com/paper/2509.16204/) (2025)
- [BigDocs: An Open Dataset for Training Multimodal Models on Document and Code Tasks](https://mlsanity.com/paper/2412.04626/) (2024)
- [EMIT: Enhancing MLLMs for Industrial Anomaly Detection via Difficulty-Aware GRPO](https://mlsanity.com/paper/2507.21619/) (2025)
- [A Unified Industrial Large Knowledge Model Framework in Industry 4.0 and Smart Manufacturing](https://mlsanity.com/paper/2312.14428/) (2023)
- [LogicQA: Logical Anomaly Detection with Vision Language Model Generated Questions](https://mlsanity.com/paper/2503.20252/) (2025)
- [GraphOmni: A Comprehensive and Extensible Benchmark Framework for Large Language Models on Graph-theoretic Tasks](https://mlsanity.com/paper/2504.12764/) (2025)
- [FailureSensorIQ: A Multi-Choice QA Dataset for Understanding Sensor Relationships and Failure Modes](https://mlsanity.com/paper/2506.03278/) (2025)
- [A VLM-based Method for Visual Anomaly Detection in Robotic Scientific Laboratories](https://mlsanity.com/paper/2506.05405/) (2025)
- [The Underappreciated Power of Vision Models for Graph Structural Understanding](https://mlsanity.com/paper/2510.24788/) (2025)
- [MME-Industry: A Cross-Industry Multimodal Evaluation Benchmark](https://mlsanity.com/paper/2501.16688/) (2025)
- [Ano-SuPs: Multi-size anomaly detection for manufactured products by identifying suspected patches](https://mlsanity.com/paper/2309.11120/) (2023)
- [F2PAD: A General Optimization Framework for Feature-Level to Pixel-Level Anomaly Detection](https://mlsanity.com/paper/2407.06519/) (2024)
- [LazyVLM: Neuro-Symbolic Approach to Video Analytics](https://mlsanity.com/paper/2505.21459/) (2025)

## Built on by (most cited first, 2 of 2)

- [IndustryBench-MIPU: Benchmarking Multi-Image Attribute Value Extraction for Industrial Products](https://mlsanity.com/paper/2606.14383/) (2026)
- [An Exam for Active Observers](https://mlsanity.com/paper/2607.16165/) (2026)

## Abstract (arXiv)

The manufacturing sector is increasingly adopting Multimodal Large Language Models (MLLMs) to transition from simple perception to autonomous execution, yet current evaluations fail to reflect the rigorous demands of real-world manufacturing environments. Progress is hindered by data scarcity and a lack of fine-grained domain semantics in existing datasets. To bridge this gap, we introduce FORGE. Wefirst construct a high-quality multimodal dataset that combines real-world 2D images and 3D point clouds, annotated with fine-grained domain semantics (e.g., exact model numbers). We then evaluate 18 state-of-the-art MLLMs across three manufacturing tasks, namely workpiece verification, structural surface inspection, and assembly verification, revealing significant performance gaps. Counter to conventional understanding, the bottleneck analysis shows that visual grounding is not the primary limiting factor. Instead, insufficient domain-specific knowledge is the key bottleneck, setting a clear direction for future research. Beyond evaluation, we show that our structured annotations can serve as an actionable training resource: supervised fine-tuning of a compact 3B-parameter model on our data yields up to 90.8% relative improvement in accuracy on held-out manufacturing scenarios, providing preliminary evidence for a practical pathway toward domain-adapted manufacturing MLLMs. The code and datasets are available at https://ai4manufacturing.github.io/forge-web.