# EngIntervene: Benchmarking Multimodal Engineering State Understanding and Design Intervention Reasoning

arXiv: https://arxiv.org/abs/2609.33261 · HTML: https://mlsanity.com/paper/2609.33261/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 27 Sep 2026.

## Builds on (most cited first, 13 of 13)

- [MMSI-Bench: A Benchmark for Multi-Image Spatial Intelligence](https://mlsanity.com/paper/2505.23764/) (2025)
- [DynaMath: A Dynamic Visual Benchmark for Evaluating Mathematical Reasoning Robustness of Vision Language Models](https://mlsanity.com/paper/2411.00836/) (2024)
- [MME-CoT: Benchmarking Chain-of-Thought in Large Multimodal Models for Reasoning Quality, Robustness, and Efficiency](https://mlsanity.com/paper/2502.09621/) (2025)
- [SeePhys: Does Seeing Help Thinking? -- Benchmarking Vision-Based Physics Reasoning](https://mlsanity.com/paper/2505.19099/) (2025)
- [DesignQA: A Multimodal Benchmark for Evaluating Large Language Models' Understanding of Engineering Documentation](https://mlsanity.com/paper/2404.07917/) (2024)
- [AnalogCoder-Pro: Unifying Analog Circuit Generation and Optimization via Multi-modal LLMs](https://mlsanity.com/paper/2508.02518/) (2025)
- [Unveiling the Compositional Ability Gap in Vision-Language Reasoning Model](https://mlsanity.com/paper/2505.19406/) (2025)
- [EngiBench: A Benchmark for Evaluating Large Language Models on Engineering Problem Solving](https://mlsanity.com/paper/2509.17677/) (2025)
- [BuildArena: A Physics-Aligned Interactive Benchmark of LLMs for Engineering Construction](https://mlsanity.com/paper/2510.16559/) (2025)
- [FEM-Bench: A Structured Scientific Reasoning Benchmark for Evaluating Code-Generating LLMs](https://mlsanity.com/paper/2512.20732/) (2025)
- [BikeBench: A Bicycle Design Benchmark for Generative Models with Objectives and Constraints](https://mlsanity.com/paper/2508.00830/) (2025)
- [CircuitSense: A Hierarchical MLLM Benchmark Bridging Visual Comprehension and Symbolic Reasoning in Engineering Design Process](https://mlsanity.com/paper/2509.22339/) (2025)
- [Rule2DRC: Benchmarking LLM Agents for DRC Script Synthesis with Execution-Guided Test Generation](https://mlsanity.com/paper/2605.15669/) (2026)

## Abstract (arXiv)

Multimodal engineering benchmarks largely evaluate static understanding, such as recognizing components, interpreting diagrams, or answering technical questions. This leaves a missing middle between engineering perception and full design generation: whether a model can use an understood system state to reason about relations, constraints, and the consequences of design changes. We introduce textscEngIntervene, a benchmark for this capability. It contains 3,229 questions across seven engineering domains and organizes evaluation into four levels: state grounding (T1), relational and mechanistic reasoning (T2), constraint-aware diagnosis (T3), and intervention reasoning (T4), which asks whether a proposed modification achieves its target while preserving required constraints. The tasks instantiate a unified engineering state representation spanning objects, relations, constraints, and design objectives, and T2--T4 are scored against structured reference answers with atomic criteria. Across open- and closed-weight multimodal models, stronger grounding does not reliably translate into better diagnosis or intervention, and the best open-weight model trails the best closed model by 14.7 percentage points on the T2--T4 average. Removing or shuffling visual evidence consistently degrades performance, while benchmark-specific supervised fine-tuning improves T1 but not T2--T4. T4 further exposes a large gap between satisfying individual revision criteria and producing a fully valid intervention. Engineering reasoning thus requires not only recovering the current state, but also reliably using it to reason about constraints and post-intervention consequences. Code and benchmark artifacts are available at https://github.com/changcv2021/EngIntervene