mlsanityData to 14 Oct 2026

VIBE: Visual Instruction Based Editor

arXiv 2601.02242 ↗First posted 5 Jan 2026cs.CV cs.AI cs.LGcited by 2

Not on the map. Why?

Abstract (arXiv)

Instruction-based image editing is among the fastest developing areas in generative AI. Over the past year, the field has reached a new level, with dozens of open-source models released alongside highly capable commercial systems. However, only a limited number of open-source approaches currently achieve real-world quality. In addition, diffusion backbones, the dominant choice for these pipelines, are often large and computationally expensive for many deployments and research settings, with widely used variants typically containing 6B to 20B parameters. This paper presents a compact, high-throughput instruction-based image editing pipeline that uses a modern 2B-parameter Qwen3-VL model to guide the editing process and the 1.6B-parameter diffusion model Sana1.5 for image generation. Our design decisions across architecture, data processing, training configuration, and evaluation target low-cost inference and strict source consistency while maintaining high quality across the major edit categories feasible at this scale. Evaluated on the ImgEdit and GEdit benchmarks, the proposed method matches or exceeds the performance of substantially heavier baselines, including models with several times as many parameters and higher inference cost, and is particularly strong on edits that require preserving the input image, such as an attribute adjustment, object removal, background edits, and targeted replacement. The model fits within 24 GB of GPU memory and generates edited images at up to 2K resolution in approximately 4 seconds on an NVIDIA H100 in BF16, without additional inference optimizations or distillation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 5 Jan 2026.

Builds on

62 of 81 references.

References · 62

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
  8. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  9. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  10. 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
  12. 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
38 more
  1. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  2. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  3. 2019Searching for MobileNetV3cited by 1,994
  4. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  5. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
  6. 2025Qwen-Image Technical Reportcited by 1,044
  7. 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
  9. 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
  10. 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
  11. 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
  12. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  13. 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
  14. 2024OmniGen: Unified Image Generationcited by 394
  15. 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
  16. 2021VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalizationcited by 287
  17. 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 277
  18. 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
  19. 2023Generating Images with Multimodal Language Modelscited by 260
  20. 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
  21. 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
  22. 2025Transfer between Modalities with MetaQueriescited by 236
  23. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 231
  24. 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
  25. 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
  26. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  27. 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 173
  28. 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
  29. 2023Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Modelscited by 153
  30. 2024OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-oncited by 151
  31. 2024DiT4Edit: Diffusion Transformer for Image Editingcited by 101
  32. 2025LongCat-Image Technical Reportcited by 100
  33. 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
  34. 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
  35. 2024ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertioncited by 71
  36. 2022HaGRID - HAnd Gesture Recognition Image Datasetcited by 43
  37. 2023MiVOLO: Multi-input Transformer for Age and Gender Estimationcited by 42
  38. 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 41

50 most cited of 62 All 62 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
  2. 2026Instruction-Based Video Editing by Repurposing an Image Editing Model
  3. 2026Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications
  4. 2025Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder
  5. 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editor
  6. 2025Image Editing As Programs with Diffusion Models
  7. 2025X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
  8. 2026Instruction-based Image Editing with Planning, Reasoning, and Generation