mlsanityData to 7 Oct 2026

Rendering-in-the-Loop: An Execution-Driven Agent for Interactive Web Development

arXiv 2609.02088 ↗First posted 2 Sep 2026cs.CVcited by 0

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Multimodal large language models have achieved remarkable progress in front-end web development, generating interactive webpages from multimodal references such as screenshots and interaction videos. However, existing work largely emphasizes visual metrics such as aesthetics and layout similarity, while overlooking the more critical validation of interactive functionality. We present RILA, an execution-driven agent that puts browser rendering in the loop, iteratively editing generated code from runtime interaction feedback. RILA introduces an Action Interaction Verification (AIV) module that replays the reference interaction trajectory on the generated webpage to collect grounded execution-aware observations, and an Execution-aware Rendering Score (ERS) that jointly measures interaction correctness and visual fidelity to guide iterative optimization. We further build an execution-verified data synthesis pipeline that produces diverse, high-quality training data, offering gains complementary to inference-time optimization. On IWR-Bench, RILA consistently improves both interaction and visual fidelity across foundation models. Notably, with our training pipeline, RILA lifts the compact Qwen3.5-9B backbone from 40.40% to 57.52%, surpassing far larger one-shot generators, including the 1T-parameter Kimi-K2.6 (55.61%) and the proprietary GPT-5.5 (55.74%).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 21 of its 30 references that are in the map. First posted on arXiv 2 Sep 2026.

Builds on

21 of 30 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials12 Dec
  2. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan

Other references · 19

  1. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,193
  2. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,154
  3. 2024OpenHands: An Open Platform for AI Software Developers as Generalist Agentscited by 956
  4. 2024Executable Code Actions Elicit Better LLM Agentscited by 537
  5. 2024Agentless: Demystifying LLM-based Software Engineering Agentscited by 400
  6. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  7. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 177
  8. 2024Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineeringcited by 146
  9. 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 140
  10. 2024Unlocking the conversion of Web Screenshots into HTML Code with the WebSight Datasetcited by 126
  11. 2017pix2code: Generating Code from a Graphical User Interface Screenshotcited by 113
  12. 2024Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMscited by 73
7 more
  1. 2025WebGen-Bench: Evaluating LLMs on Generating Interactive and Functional Websites from Scratchcited by 66
  2. 2024Interaction2Code: Benchmarking MLLM-based Interactive Webpage Code Generation from Interactive Prototypingcited by 50
  3. 2024Sketch2Code: Evaluating Vision-Language Models for Interactive Web Design Prototypingcited by 29
  4. 2025UI2Code^N: UI-to-Code Generation as Interactive Visual Optimizationcited by 12
  5. 2025IWR-Bench: Can LVLMs reconstruct interactive webpage from a user interaction video?cited by 10
  6. 2026VisRefiner: Learning from Visual Differences for Screenshot-to-Code Generationcited by 7
  7. 2026Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpagescited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026InteractWeb-Bench: Can Multimodal Agent Escape Blind Execution in Interactive Website Generation?
  2. 2026UI2App: Benchmarking Visual Interaction Inference in Executable Web Application Generation
  3. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
  4. 2025WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning
  5. 2025FronTalk: Benchmarking Front-End Development as Conversational Code Generation with Multi-Modal Feedback
  6. 2026WebCompass: Towards Multimodal Web Coding Evaluation for Code Language Models
  7. 2026Vision2Web: A Hierarchical Benchmark for Visual Website Development with Agent Verification
  8. 2025WebGen-Agent: Enhancing Interactive Website Generation with Multi-Level Feedback and Step-Level Reinforcement Learning