# FigmaTrace: Capturing Creative Nuances in Human Figma Design Workflows

arXiv: https://arxiv.org/abs/2608.21460 · HTML: https://mlsanity.com/paper/2608.21460/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 26 of its 30 references that are in the map.
- First posted on arXiv 20 Aug 2026.

## Placed in

- Multimodal LLMs / vision-language models / [VLM benchmarks and visual reasoning](https://mlsanity.com/mllm/vlm-benchmarks-and-visual-reasoning/)

## Builds on (most cited first, 26 of 26)

- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuning](https://mlsanity.com/paper/2408.05517/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis](https://mlsanity.com/paper/2505.13227/) (2025)
- [Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Dataset](https://mlsanity.com/paper/2403.09029/) (2024)
- [Large Multimodal Agents: A Survey](https://mlsanity.com/paper/2402.15116/) (2024)
- [Q-Insight: Understanding Image Quality via Visual Reinforcement Learning](https://mlsanity.com/paper/2503.22679/) (2025)
- [VisualQuality-R1: Reasoning-Induced Image Quality Assessment via Reinforcement Learning to Rank](https://mlsanity.com/paper/2505.14460/) (2025)
- [A Survey on (M)LLM-Based GUI Agents](https://mlsanity.com/paper/2504.13865/) (2025)
- [Pure Vision Language Action (VLA) Models: A Comprehensive Survey](https://mlsanity.com/paper/2509.19012/) (2025)
- [VideoGUI: A Benchmark for GUI Automation from Instructional Videos](https://mlsanity.com/paper/2406.10227/) (2024)
- [VideoAgentTrek: Computer Use Pretraining from Unlabeled Videos](https://mlsanity.com/paper/2510.19488/) (2025)
- [Evaluating Vision-Language Models for Emotion Recognition](https://mlsanity.com/paper/2502.05660/) (2025)
- [A Survey on MLLM-based Visually Rich Document Understanding: Methods, Challenges, and Emerging Trends](https://mlsanity.com/paper/2507.09861/) (2025)
- [DesignPref: Capturing Personal Preferences in Visual Design Generation](https://mlsanity.com/paper/2511.20513/) (2025)
- [Can Vision Language Models Assess Graphic Design Aesthetics? A Benchmark, Evaluation, and Dataset Perspective](https://mlsanity.com/paper/2603.01083/) (2026)
- [Humor in Pixels: Benchmarking Large Multimodal Models Understanding of Online Comics](https://mlsanity.com/paper/2509.12248/) (2025)
- [Document Intelligence in the Era of Large Language Models: A Survey](https://mlsanity.com/paper/2510.13366/) (2025)
- [CANVAS: A Benchmark for Vision-Language Models on Tool-Based User Interface Design](https://mlsanity.com/paper/2511.20737/) (2025)
- [LH-Bench: Skill-Grounded Evaluation of Long-Horizon Agents on Subjective Enterprise Tasks](https://mlsanity.com/paper/2603.22744/) (2026)
- [Figma2Code: Automating Multimodal Design to Code in the Wild](https://mlsanity.com/paper/2604.13648/) (2026)
- [CoGen: Creation of Reusable UI Components in Figma via Textual Commands](https://mlsanity.com/paper/2601.10536/) (2026)
- [I Came, I Saw, I Explained: Benchmarking Multimodal LLMs on Figurative Meaning in Memes](https://mlsanity.com/paper/2603.23229/) (2026)

## Abstract (arXiv)

Vision Language Models have recently shown improvements in several objective and verifiable domains such as object detection but continue to underperform on subjective and creative design tasks. A major contributor to this performance gap is the lack of high quality human workflow data that captures a diverse set of preferences and decisions that make human experts good at design tasks. In this work, we first define a unique, expert curated taxonomy of design skills and best practices which we further expand into a set of 126 open ended, subjective, long horizon tasks. Built on top of this and expert solutions, our dataset FigmaTrace contains over 200 hours of human captured video data converted into 3469 design trajectories using a novel design phase-based method. We use our dataset to train four models and show that training on FigmaTrace leads to a performance improvement comparable to frontier closed models such as textscClaude-Opus-5 and textscGPT-5.6-Sol on four out of distribution agentic GUI environments. We further perform a useful ablation to attribute these performance improvements to a design phase-based video to trajectory conversion which outperforms prior length-based conversion approaches. Finally, we perform a qualitative analysis on the best performing textscQwen3.8-27B outputs to better correlate performance improvements to FigmaTrace's trends. We open source our dataset and the best model for the community.