mlsanityData to 7 Oct 2026

EditCaption: Human-Refined SFT and HAE-DPO for Image Editing Instruction Synthesis

arXiv 2604.08213 ↗First posted 9 Apr 2026cs.CV cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Diffusion and multimodal model alignment

also in 1 more
Abstract (arXiv)

High-quality source-target image pairs with precise editing instructions are essential for instruction-guided image editing, yet constructing such training triplets at scale remains costly. Recent pipelines often rely on vision-language models to synthesize editing instructions automatically, but we find that strong VLMs still struggle to describe visual transformations between image pairs. In particular, they exhibit three recurring failure modes: orientation inconsistency, viewpoint ambiguity, and missing fine-grained attributes. In a human evaluation on 400 image pairs, several open-source VLM baselines produce critical-error rates above 47%, making many synthesized instructions unsuitable for downstream training. To address this, we propose EditCaption, a two-stage post-training pipeline for image editing instruction synthesis. First, we construct a 100K supervised fine-tuning dataset through GLM-based auto-captioning, EditScore filtering, and human refinement. Second, we collect 10K human-annotated preference pairs, where each rejected instruction is labeled with its primary error type and severity. Based on this dataset, we propose Hardness-Adaptive Error-Aware DPO (HAE-DPO), a task-adapted DPO objective that introduces an adaptive margin based on human-labeled severity, failure-mode type, and reference-model hardness. Experiments across three benchmarks demonstrate that our 235B model with SFT+HAE-DPO achieves state-of-the-art performance among open-source and closed models, scoring 4.720 on Eval-400, 4.672 on HQ-Edit, and 4.651 on ByteMorph-Bench -- surpassing Gemini-3-Pro on all three. Human evaluation confirms critical error rates drop from 47.75% to 17.50%, with correct rates improving from 41.75% to 70.25%, surpassing Gemini-3-Pro (66.00%).

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and multimodal model alignment, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 13 of its 19 references that are in the map. First posted on arXiv 9 Apr 2026.

Builds on

13 of 19 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modeling28 Sep

Other references · 12

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  3. 2023InstructBLIP: Towards General-purpose Vision-Language Models with Instruction Tuningcited by 2,855
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,591
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  6. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Modelscited by 1,172
  9. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  10. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 417
  11. 2025ByteMorph: Benchmarking Instruction-Guided Image Editing with Non-Rigid Motionscited by 19
  12. 2026Instruction-based Image Editing with Planning, Reasoning, and Generationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024InsightEdit: Towards Better Instruction Following for Image Editing
  2. 2026GEditBench v2: A Human-Aligned Benchmark for General Image Editing
  3. 2026Human-Aligned MLLM Judges for Fine-Grained Image Editing Evaluation: A Benchmark, Framework, and Analysis
  4. 2025SuperEdit: Rectifying and Facilitating Supervision for Instruction-Based Image Editing
  5. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Models
  6. 2025EditReward: A Human-Aligned Reward Model for Instruction-Guided Image Editing
  7. 2025ADIEE: Automatic Dataset Creation and Scorer for Instruction-Guided Image Editing Evaluation
  8. 2026Scalable Minimal-Change Learning for Controllable Image Editing