EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis
On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment
also in 1 more
Abstract (arXiv)
High-quality source-target image pairs with precise editing instructions are essential for instruction-guided image editing, yet constructing such training triplets at scale remains costly. Recent pipelines often rely on vision-language models to synthesize editing instructions automatically, but we find that strong VLMs still struggle to describe visual transformations between image pairs. In particular, they exhibit three recurring failure modes: orientation inconsistency, viewpoint ambiguity, and missing fine-grained attributes. In a human evaluation on 400 image pairs, several open-source VLM baselines produce critical-error rates above 47%, making many synthesized instructions unsuitable for downstream training. To address this, we propose EditCaption, a two-stage post-training pipeline for image editing instruction synthesis. First, we construct a 100K supervised fine-tuning dataset through GLM-based auto-captioning, EditScore filtering, and human refinement. Second, we collect 10K human-annotated preference pairs, where each rejected instruction is labeled with its primary error type and severity. Based on this dataset, we propose Hardness-Adaptive Error-Aware DPO (HAE-DPO), a task-adapted DPO objective that introduces an adaptive margin based on human-labeled severity, failure-mode type, and reference-model hardness. Experiments across three benchmarks demonstrate that our 235B model with SFT+HAE-DPO achieves state-of-the-art performance among open-source and closed models, scoring 4.720 on Eval-400, 4.672 on HQ-Edit, and 4.651 on ByteMorph-Bench -- surpassing Gemini-3-Pro on all three. Human evaluation confirms critical error rates drop from 47.75% to 17.50%, with correct rates improving from 41.75% to 70.25%, surpassing Gemini-3-Pro (66.00%).
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 19 references that are in the map. First posted on arXiv 9 Apr 2026.
Builds on
13 of 19 references.
Closest ancestors · 1
Same direction, newest first
Other references · 12
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
- 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
- 2025ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motionscited by 19
- 2026Instruction-based Image Editing with Planning, Reasoning, and Generationcited by 4
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024InsightEdit: Towards Better Instruction Following for Image Editing
- 2026GEditBench v2: A Human-Aligned Benchmark for General Image Editing
- 2026Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
- 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing
- 2023Guiding Instruction-based Image Editing via Multimodal Large Language Models
- 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing
- 2025ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation
- 2026Scalable Minimal-Change Learning for Controllable Image Editing