# SMART-Editor: A Multi-Agent Framework for Human-Like Design Editing with Structural Integrity

arXiv: https://arxiv.org/abs/2507.23095 · HTML: https://mlsanity.com/paper/2507.23095/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 34 of its 45 references that are in the map.
- First posted on arXiv 30 Jul 2025.

## Builds on (most cited first, 30 of 34)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [InCoder: A Generative Model for Code Infilling and Synthesis](https://mlsanity.com/paper/2204.05999/) (2022)
- [DragDiffusion: Harnessing Diffusion Models for Interactive Point-based Image Editing](https://mlsanity.com/paper/2306.14435/) (2023)
- [Drag Your GAN: Interactive Point-based Manipulation on the Generative Image Manifold](https://mlsanity.com/paper/2305.10973/) (2023)
- [HQ-Edit: A High-Quality Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2404.09990/) (2024)
- [HIVE: Harnessing Human Feedback for Instructional Visual Editing](https://mlsanity.com/paper/2303.09618/) (2023)
- [Design2Code: Benchmarking Multimodal Code Generation for Automated Front-End Engineering](https://mlsanity.com/paper/2403.03163/) (2024)
- [LayoutDM: Discrete Diffusion Model for Controllable Layout Generation](https://mlsanity.com/paper/2303.08137/) (2023)
- [GenArtist: Multimodal LLM as an Agent for Unified Image Generation and Editing](https://mlsanity.com/paper/2407.05600/) (2024)
- [LayoutGAN: Generating Graphic Layouts with Wireframe Discriminators](https://mlsanity.com/paper/1901.06767/) (2019)
- [Grounding DINO 1.5: Advance the "Edge" of Open-Set Object Detection](https://mlsanity.com/paper/2405.10300/) (2024)
- [LayoutVAE: Stochastic Scene Layout Generation From a Label Set](https://mlsanity.com/paper/1907.10719/) (2019)
- [SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editing](https://mlsanity.com/paper/2405.04007/) (2024)
- [InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions](https://mlsanity.com/paper/2305.18047/) (2023)
- [LayoutDM: Transformer-based Diffusion Model for Layout Generation](https://mlsanity.com/paper/2305.02567/) (2023)
- [LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts](https://mlsanity.com/paper/2310.10640/) (2023)
- [COLE: A Hierarchical Generation Framework for Multi-Layered and Editable Graphic Design](https://mlsanity.com/paper/2311.16974/) (2023)
- [PosterCraft: Rethinking High-Quality Aesthetic Poster Generation in a Unified Framework](https://mlsanity.com/paper/2506.10741/) (2025)
- [Text-Driven Image Editing via Learnable Regions](https://mlsanity.com/paper/2311.16432/) (2023)
- [HumanEdit: A High-Quality Human-Rewarded Dataset for Instruction-based Image Editing](https://mlsanity.com/paper/2412.04280/) (2024)
- [CreatiPoster: Towards Editable and Controllable Multi-Layer Graphic Design Generation](https://mlsanity.com/paper/2506.10890/) (2025)
- [PosterO: Structuring Layout Trees to Enable Language Models in Generalized Content-Aware Layout Generation](https://mlsanity.com/paper/2505.07843/) (2025)
- [SciPostLayout: A Dataset for Layout Analysis and Layout Generation of Scientific Posters](https://mlsanity.com/paper/2407.19787/) (2024)
- [Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models](https://mlsanity.com/paper/2404.15271/) (2024)

## Abstract (arXiv)

We present SMART-Editor, a framework for compositional layout and content editing across structured (posters, websites) and unstructured (natural images) domains. Unlike prior models that perform local edits, SMART-Editor preserves global coherence through two strategies: Reward-Refine, an inference-time rewardguided refinement method, and RewardDPO, a training-time preference optimization approach using reward-aligned layout pairs. To evaluate model performance, we introduce SMARTEdit-Bench, a benchmark covering multi-domain, cascading edit scenarios. SMART-Editor outperforms strong baselines like InstructPix2Pix and HIVE, with RewardDPO achieving up to 15% gains in structured settings and Reward-Refine showing advantages on natural images. Automatic and human evaluations confirm the value of reward-guided planning in producing semantically consistent and visually aligned edits.