# FormFactory: An Interactive Benchmarking Suite for Multimodal Form-Filling Agents

arXiv: https://arxiv.org/abs/2506.01520 · HTML: https://mlsanity.com/paper/2506.01520/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 21 of its 49 references that are in the map.
- First posted on arXiv 2 Jun 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 21 of 21)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [Mind2Web: Towards a Generalist Agent for the Web](https://mlsanity.com/paper/2306.06070/) (2023)
- [LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attention](https://mlsanity.com/paper/2303.16199/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [NExT-GPT: Any-to-Any Multimodal LLM](https://mlsanity.com/paper/2309.05519/) (2023)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [MM-REACT: Prompting ChatGPT for Multimodal Reasoning and Action](https://mlsanity.com/paper/2303.11381/) (2023)
- [Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Control](https://mlsanity.com/paper/2306.07863/) (2023)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Android in the Zoo: Chain-of-Action-Thought for GUI Agents](https://mlsanity.com/paper/2403.02713/) (2024)
- [AutoWebGLM: A Large Language Model-based Web Navigating Agent](https://mlsanity.com/paper/2404.03648/) (2024)
- [AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents](https://mlsanity.com/paper/2407.17490/) (2024)
- [A data-driven approach for learning to control computers](https://mlsanity.com/paper/2202.08137/) (2022)
- [META-GUI: Towards Multi-modal Conversational Agents on Mobile GUI](https://mlsanity.com/paper/2205.11029/) (2022)
- [SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation](https://mlsanity.com/paper/2410.15164/) (2024)
- [TabPedia: Towards Comprehensive Visual Table Understanding with Concept Synergy](https://mlsanity.com/paper/2406.01326/) (2024)
- [Multimodal Table Understanding](https://mlsanity.com/paper/2406.08100/) (2024)
- [VideoGUI: A Benchmark for GUI Automation from Instructional Videos](https://mlsanity.com/paper/2406.10227/) (2024)
- [GUI Testing Arena: A Unified Benchmark for Advancing Autonomous GUI Testing Agent](https://mlsanity.com/paper/2412.18426/) (2024)
- [2D-TPE: Two-Dimensional Positional Encoding Enhances Table Understanding for Large Language Models](https://mlsanity.com/paper/2409.19700/) (2024)

## Built on by (most cited first, 7 of 7)

- [WebTestPilot: Agentic End-to-End Web Testing against Natural Language Specification by Inferring Oracles with Symbolized GUI Elements](https://mlsanity.com/paper/2602.11724/) (2026)
- [Taming Actor-Observer Asymmetry in Agents via Dialectical Alignment](https://mlsanity.com/paper/2604.19548/) (2026)
- [WARC-Bench: Web Archive Based Benchmark for GUI Subtask Executions](https://mlsanity.com/paper/2510.09872/) (2025)
- [EO-Gym: A Multimodal, Interactive Environment for Earth Observation Agents](https://mlsanity.com/paper/2605.01250/) (2026)
- [Agent Safety Is Action Alignment](https://mlsanity.com/paper/2606.28739/) (2026)
- [CLAIRE: A Schema-Grounded Hybrid Workflow for Healthcare Administrative Form Completion](https://mlsanity.com/paper/2609.32787/) (2026)
- [CRAFT: An Agentic Spreadsheet Form Filling System with Template Awareness](https://mlsanity.com/paper/2610.04437/) (2026)

## Abstract (arXiv)

Online form filling is a common yet labor-intensive task involving extensive keyboard and mouse interactions. Despite the long-standing vision of automating this process with "one click", existing tools remain largely rule-based and lack generalizable, generative capabilities. Recent advances in Multimodal Large Language Models (MLLMs) have enabled promising agents for GUI-related tasks in general-purpose scenarios. However, they struggle with the unique challenges of form filling, such as flexible layouts and the difficulty of aligning textual instructions with on-screen fields. To bridge this gap, we formally define the form-filling task and propose FormFactory, an interactive benchmarking suite comprising a web-based interface, backend evaluation module, and carefully constructed dataset. Our benchmark covers diverse real-world scenarios, incorporates various field formats, and simulates high-fidelity form interactions. We conduct a comprehensive evaluation of state-of-the-art MLLMs and observe that no model surpasses 5% accuracy, underscoring the inherent difficulty of the task. These findings also reveal significant limitations in current models' visual layout reasoning and field-value alignment abilities. We hope our benchmark can serve as a stepping stone for further research into robust, practical form-filling agents.