Benchmarking and Mechanistic Analysis of Vision-Language Models for Cross-Depiction Assembly Instruction Alignment
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
Abstract (arXiv)
2D assembly diagrams are often abstract and hard to follow, creating a need for intelligent assistants that can monitor progress, detect errors, and provide step-by-step guidance. In mixed reality settings, such systems must recognize completed and ongoing steps from the camera feed and align them with the diagram instructions. Vision Language Models (VLMs) show promise for this task, but face a depiction gap because assembly diagrams and video frames share few visual features. To systematically assess this gap, we construct IKEA-Bench, a benchmark of 1,623 questions across 6 task types on 29 IKEA furniture products, and evaluate 19 VLMs (2B-38B) under three alignment strategies. Our key findings: (1) assembly instruction understanding is recoverable via text, but text simultaneously degrades diagram-to-video alignment; (2) architecture family predicts alignment accuracy more strongly than parameter count; (3) video understanding remains a hard bottleneck unaffected by strategy. A three-level mechanistic analysis further reveals that diagrams and video occupy disjoint ViT subspaces, and that adding text shifts models from visual to text-driven reasoning. These results identify visual encoding as the primary target for improving cross-depiction robustness. Project page: https://ryenhails.github.io/IKEA-Bench/
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 31 references that are in the map. First posted on arXiv 1 Apr 2026.
Builds on
16 of 31 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2019Similarity of Neural Network Representations Revisitedcited by 1,695
- 2016Understanding intermediate layers using linear classifier probescited by 1,279
- 2022Assembly101: A Large-Scale Multi-View Video Dataset for Understanding Procedural Activitiescited by 284
- 2023HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real Worldcited by 159
- 2020The IKEA ASM Dataset: Understanding People Assembling Furniture through Actions, Objects and Posecited by 95
- 2025The Hidden Life of Tokens: Reducing Hallucination of Large Vision-Language Models via Visual Information Steeringcited by 62
- 2023CaptainCook4D: A Dataset for Understanding Errors in Procedural Activitiescited by 47
- 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
- 2023IKEA-Manual: Seeing Shape Assembly Step by Stepcited by 22
- 2025Manual2Skill: Learning to Read Manuals and Acquire Robotic Skills for Furniture Assembly Using Vision-Language Modelscited by 13
- 2024IKEA Manuals at Work: 4D Grounding of Assembly Instructions on Internet Videoscited by 11
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026AssemState: Manual and Physical-State-Guided Reasoning for Zero-shot Furniture Assembly
- 2026Flat-Pack Bench: Evaluating Spatio-Temporal Understanding in Large Vision-Language Models through Furniture Assembly
- 2025Manual2Skill++: Connector-Aware General Robotic Assembly from Instruction Manuals via Vision-Language Models
- 2023Evaluating Robustness of Visual Representations for Object Assembly Task Requiring Spatio-Geometrical Reasoning
- 2024Comparison Visual Instruction Tuning
- 2026Shape of Thought: Progressive Object Assembly via Visual Chain-of-Thought
- 2026Brick-Composer: Using MLLMs for Assembly with Diverse Bricks
- 2026Enginuity: A Dataset and Benchmark for Vision-Language Understanding of Engineering Diagrams