# ReLook: Vision-Grounded RL with a Multimodal LLM Critic for Agentic Web Coding

arXiv: https://arxiv.org/abs/2510.11498 · HTML: https://mlsanity.com/paper/2510.11498/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 29 of its 34 references that are in the map.
- First posted on arXiv 13 Oct 2025.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/)

## Builds on (most cited first, 29 of 29)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Reflexion: Language Agents with Verbal Reinforcement Learning](https://mlsanity.com/paper/2303.11366/) (2023)
- [Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolution](https://mlsanity.com/paper/2409.12191/) (2024)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [Competition-Level Code Generation with AlphaCode](https://mlsanity.com/paper/2203.07814/) (2022)
- [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023)
- [CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning](https://mlsanity.com/paper/2207.01780/) (2022)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [LLM Critics Help Catch LLM Bugs](https://mlsanity.com/paper/2407.00215/) (2024)
- [Web2Code: A Large-scale Webpage-to-Code Dataset and Evaluation Framework for Multimodal LLMs](https://mlsanity.com/paper/2406.20098/) (2024)
- [ArtifactsBench: Bridging the Visual-Interactive Gap in LLM Code Generation Evaluation](https://mlsanity.com/paper/2507.04952/) (2025)
- [Web-Bench: A LLM Code Benchmark Based on Web Standards and Frameworks](https://mlsanity.com/paper/2505.07473/) (2025)
- [MLLM-Bench: Evaluating Multimodal LLMs with Per-sample Criteria](https://mlsanity.com/paper/2311.13951/) (2023)
- [UICoder: Finetuning Large Language Models to Generate User Interface Code through Automated Feedback](https://mlsanity.com/paper/2406.07739/) (2024)
- [Think-J: Learning to Think for Generative LLM-as-a-Judge](https://mlsanity.com/paper/2505.14268/) (2025)
- [Pix2Code: Learning to Compose Neural Visual Concepts as Programs](https://mlsanity.com/paper/2402.08280/) (2024)
- [CriticLean: Critic-Guided Reinforcement Learning for Mathematical Formalization](https://mlsanity.com/paper/2507.06181/) (2025)
- [CodeCriticBench: A Holistic Code Critique Benchmark for Large Language Models](https://mlsanity.com/paper/2502.16614/) (2025)
- [DesignCoder: Hierarchy-Aware and Self-Correcting UI Code Generation with Large Language Models](https://mlsanity.com/paper/2506.13663/) (2025)
- [Beyond Bounding Box: Multimodal Knowledge Learning for Object Detection](https://mlsanity.com/paper/2205.04072/) (2022)
- [HiPO: Hybrid Policy Optimization for Dynamic Reasoning in LLMs](https://mlsanity.com/paper/2509.23967/) (2025)

## Built on by (most cited first, 14 of 14)

- [NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents](https://mlsanity.com/paper/2512.12730/) (2025)
- [Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges](https://mlsanity.com/paper/2604.13602/) (2026)
- [Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs](https://mlsanity.com/paper/2511.19773/) (2025)
- [From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligence](https://mlsanity.com/paper/2511.18538/) (2025)
- [SWE-Compass: Towards Unified Evaluation of Agentic Coding Abilities for Large Language Models](https://mlsanity.com/paper/2511.05459/) (2025)
- [Beyond NL2Code: A Structured Survey of Multimodal Code Intelligence](https://mlsanity.com/paper/2606.15932/) (2026)
- [Vision-Guided Iterative Refinement for Frontend Code Generation](https://mlsanity.com/paper/2604.05839/) (2026)
- [Training and Agentic Inference Strategies for LLM-based Manim Animation Generation](https://mlsanity.com/paper/2604.18364/) (2026)
- [PaperFit: Vision-in-the-Loop Typesetting Optimization for Scientific Documents](https://mlsanity.com/paper/2605.10341/) (2026)
- [Self-Distillation Policy Optimization via Visual Feedback: Bridging Code and Visual Artifacts](https://mlsanity.com/paper/2606.10334/) (2026)
- [WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader](https://mlsanity.com/paper/2608.06474/) (2026)
- [PlanPO: Group Planning-Aware Policy Optimization for Multi-Turn Agentic LLMs](https://mlsanity.com/paper/2608.17289/) (2026)
- [WatchPoint: Executable User Feedback for Real-World Agentic Web Development](https://mlsanity.com/paper/2609.26204/) (2026)
- [AesCode: Aesthetic Code Generation with Decoupled Cross-Modal Rewards](https://mlsanity.com/paper/2610.18860/) (2026)

## Abstract (arXiv)

While Large Language Models (LLMs) excel at algorithmic code generation, they struggle with front-end development, where correctness is judged on rendered pixels and interaction. We present ReLook, an agentic, vision-grounded reinforcement learning framework that empowers an agent to close a robust generate--diagnose--refine loop by invoking a multimodal LLM (MLLM) as a tool. During training, the agent uses the MLLM-in-the-loop both as a visual critic--scoring code with screenshots--and as a source of actionable, vision-grounded feedback; a strict zero-reward rule for invalid renders anchors renderability and prevents reward hacking. To prevent behavioral collapse, we introduce Forced Optimization, a strict acceptance rule that admits only improving revisions, yielding monotonically better trajectories. At inference, we decouple the critic and run a lightweight, critic-free self-edit cycle, keeping latency comparable to base decoding while retaining most of the gains. Across three widely used benchmarks, ReLook consistently outperforms strong baselines in vision-grounded front-end code generation, highlighting the benefits of agentic perception, visual rewards, and training-inference decoupling.