mlsanityData to 14 Oct 2026

FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows

arXiv 2608.21460 ↗First posted 20 Aug 2026cs.CV cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as textscClaude-Opus-5 and textscGPT-5.6-Sol on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing textscQwen3.8-27B outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 26 of its 30 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

26 of 30 references.

References · 26

  1. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  2. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 355
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
  4. 2024On the Effects of Data Scale on UI Control Agentscited by 194
  5. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  6. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 152
  7. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  8. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 129
  9. 2024Large Multimodal Agents: A Surveycited by 105
  10. 2025Q-Insight: Understanding Image Quality via Visual Reinforcement Learningcited by 98
  11. 2025VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rankcited by 79
  12. 2025A Survey on (M)LLM-Based GUI Agentscited by 62
14 more
  1. 2025Pure Vision Language Action (VLA) Models: A Comprehensive Surveycited by 49
  2. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  3. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  4. 2025Evaluating Vision-Language Models for Emotion Recognitioncited by 19
  5. 2025A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trendscited by 8
  6. 2025DesignPref: Capturing Personal Preferences in Visual Design Generationcited by 7
  7. 2026Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspectivecited by 6
  8. 2025Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comicscited by 5
  9. 2025Document Intelligence in the Era of Large Language Models: A Surveycited by 5
  10. 2025CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Designcited by 3
  11. 2026LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Taskscited by 3
  12. 2026Figma2Code: Automating Multimodal Design to Code in the Wildcited by 3
  13. 2026CoGen: Creation of Reusable UI Components in Figma via Textual Commandscited by 2
  14. 2026I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026PainterBench: A Figural Divergent-Thinking Benchmark for Tool-Using Language Models
  2. 2026Toward a Machine Bertin: Why Visualization Needs Design Principles for Machine Cognition
  3. 2025GraphicBench: A Planning Benchmark for Graphic Design with Language Agents
  4. 2026CREATE: Creativity Evaluation of VLMs using the Visual Remote Associates Test
  5. 2026Can Vision-Language Models Analyze Human-Centered Video? Mapping Model Capabilities and Human-AI Collaborative Workflows
  6. 2026Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training
  7. 2026COMFYCLAW: Self-Evolving Skill Harnesses for Image Generation Workflows
  8. 2026DeskCraft: Benchmarking Desktop Agents on Professional Workflows and Human-in-the-Loop Collaboration