mlsanityData to 14 Oct 2026

Learning to Detect UI Principle Violations via Reinforcement Learning

arXiv 2607.20690 ↗First posted 22 Jul 2026cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Chart understanding and anomaly detection

also in 2 more
Abstract (arXiv)

Small language models and coding agents increasingly generate web front-end code, yet their outputs are typically evaluated primarily for functional correctness. A generated interface may compile, render, and pass unit tests while still violating established interface quality principles, including accessibility barriers, deceptive design patterns, poor visual hierarchy, and excessive decision complexity. Existing auditing approaches face a trade-off between cost, coverage, and scalability: expert human review provides rich judgment but is slow and expensive; frontier vision-language models offer broader reasoning capabilities but remain costly to deploy at scale; and rule-based tools such as axe-core and Lighthouse are inexpensive but primarily capture mechanically checkable accessibility issues. We investigate whether a lightweight vision-language model can serve as an effective critic for generated interfaces. We unify 19 interface-quality principles from three complementary sources of HCI knowledge: WCAG 2.2 accessibility standards, deceptive design taxonomies, and established theories of perception, cognition, and interaction. To train this critic, we construct a verified dataset of approximately 10,000 generated web pages by synthetically injecting known violations into clean, LLM-generated Tailwind pages. Continued reinforcement learning on a 4B vision-language model improves micro-F1 from 36% to 84%, with 13 of 19 principles exceeding 80% F1. The resulting critic can audit generated interfaces, filter low-quality interface training data, and provide a reward signal for design-aware code generation. We release our data-generation recipe and injection/verification prompts to support reproducible evaluation and future work on scalable interface-quality assessment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Chart understanding and anomaly detection, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 5 of its 36 references that are in the map. First posted on arXiv 22 Jul 2026.

Builds on

5 of 36 references.

References · 5

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Trainingcited by 18
  3. 2023An Ontology of Dark Patterns Knowledge: Foundations, Definitions, and a Pathway for Shared Knowledge-Buildingcited by 14
  4. 2023AidUI: Toward Automated Recognition of Dark Patterns in User Interfacescited by 9
  5. 2025A11yn: Aligning LLMs for Web Accessibility-Aware UI Generationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025WebRenderBench: Enhancing Web Interface Generation through Layout-Style Consistency and Reinforcement Learning
  2. 2026Agentic Web Accessibility Auditing: A Criterion-Specific Framework for Translating WCAG Requirements into Assessments
  3. 2025WebVIA: A Web-based Vision-Language Agentic Framework for Interactive and Verifiable UI-to-Code Generation
  4. 2025SpecifyUI: Supporting Iterative UI Design Intent Expression through Structured Specifications and Generative AI
  5. 2024UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback
  6. 2025WebAccessVL: Violation-Aware VLM for Web Accessibility
  7. 2026WebRISE: Requirement-Induced State Evaluation for MLLM-Generated Web Artifacts
  8. 2025ScreenCoder: Advancing Visual-to-Code Generation for Front-End Automation via Modular Multimodal Agents