VIBE: Visual Instruction Based Editor
Not on the map. Why?
Abstract (arXiv)
Instruction-based image editing is among the fastest developing areas in generative AI. Over the past year, the field has reached a new level, with dozens of open-source models released alongside highly capable commercial systems. However, only a limited number of open-source approaches currently achieve real-world quality. In addition, diffusion backbones, the dominant choice for these pipelines, are often large and computationally expensive for many deployments and research settings, with widely used variants typically containing 6B to 20B parameters. This paper presents a compact, high-throughput instruction-based image editing pipeline that uses a modern 2B-parameter Qwen3-VL model to guide the editing process and the 1.6B-parameter diffusion model Sana1.5 for image generation. Our design decisions across architecture, data processing, training configuration, and evaluation target low-cost inference and strict source consistency while maintaining high quality across the major edit categories feasible at this scale. Evaluated on the ImgEdit and GEdit benchmarks, the proposed method matches or exceeds the performance of substantially heavier baselines, including models with several times as many parameters and higher inference cost, and is particularly strong on edits that require preserving the input image, such as an attribute adjustment, object removal, background edits, and targeted replacement. The model fits within 24 GB of GPU memory and generates edited images at up to 2K resolution in approximately 4 seconds on an NVIDIA H100 in BF16, without additional inference optimizations or distillation.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 62 of its 81 references that are in the map. First posted on arXiv 5 Jan 2026.
Builds on
62 of 81 references.
References · 62
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2023GPT-4 Technical Reportcited by 18,396
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,503
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2024Scaling Rectified Flow Transformers for High-Resolution Image Synthesiscited by 4,181
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,152
- 2022InstructPix2Pix: Learning to Follow Image Editing Instructionscited by 2,611
38 more
- 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
- 2019Searching for MobileNetV3cited by 1,994
- 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
- 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,179
- 2025Qwen-Image Technical Reportcited by 1,044
- 2025FLUX.1 Kontext: Flow Matching for In-Context Image Generation and Editing in Latent Spacecited by 1,006
- 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 906
- 2025Emerging Properties in Unified Multimodal Pretrainingcited by 897
- 2023Diffusion Model Alignment Using Direct Preference Optimizationcited by 790
- 2023MasaCtrl: Tuning-Free Mutual Self-Attention Control for Consistent Image Synthesis and Editingcited by 727
- 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
- 2025Step1X-Edit: A Practical Framework for General Image Editingcited by 429
- 2024OmniGen: Unified Image Generationcited by 394
- 2024SANA: Efficient High-Resolution Image Synthesis with Linear Diffusion Transformerscited by 340
- 2021VITON-HD: High-Resolution Virtual Try-On via Misalignment-Aware Normalizationcited by 287
- 2025Z-Image: An Efficient Image Generation Foundation Model with Single-Stream Diffusion Transformercited by 277
- 2024UltraEdit: Instruction-based Fine-Grained Image Editing at Scalecited by 268
- 2023Generating Images with Multimodal Language Modelscited by 260
- 2025OmniGen2: Towards Instruction-Aligned Multimodal Generationcited by 253
- 2024AnyEdit: Mastering Unified High-Quality Image Editing for Any Ideacited by 241
- 2025Transfer between Modalities with MetaQueriescited by 236
- 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 231
- 2025In-Context Edit: Enabling Instructional Image Editing with In-Context Generation in Large Scale Diffusion Transformercited by 204
- 2024HQ-Edit: A High-Quality Dataset for Instruction-based Image Editingcited by 193
- 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
- 2023HIVE: Harnessing Human Feedback for Instructional Visual Editingcited by 173
- 2025SANA 1.5: Efficient Scaling of Training-Time and Inference-Time Compute in Linear Diffusion Transformercited by 165
- 2023Concept Sliders: LoRA Adaptors for Precise Control in Diffusion Modelscited by 153
- 2024OOTDiffusion: Outfitting Fusion based Latent Diffusion for Controllable Virtual Try-oncited by 151
- 2024DiT4Edit: Diffusion Transformer for Image Editingcited by 101
- 2025LongCat-Image Technical Reportcited by 100
- 2024SEED-Data-Edit Technical Report: A Hybrid Dataset for Instructional Image Editingcited by 97
- 2025ACE++: Instruction-Based Image Creation and Editing via Context-Aware Content Fillingcited by 95
- 2024ObjectDrop: Bootstrapping Counterfactuals for Photorealistic Object Removal and Insertioncited by 71
- 2022HaGRID - HAnd Gesture Recognition Image Datasetcited by 43
- 2023MiVOLO: Multi-input Transformer for Age and Gender Estimationcited by 42
- 2025Calibrated Multi-Preference Optimization for Aligning Diffusion Modelscited by 41
50 most cited of 62 All 62 (JSON)
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2026How Well Do Models Follow Visual Instructions? VIBE: A Systematic Benchmark for Visual Instruction-Driven Image Editing
- 2026Instruction-Based Video Editing by Repurposing an Image Editing Model
- 2026Instruction-based Image Editing: A Survey on Data, Models, Evaluation, and Applications
- 2025Lego-Edit: A General Image Editing Framework with Model-Level Bricks and MLLM Builder
- 2025EditThinker: Unlocking Iterative Reasoning for Any Image Editor
- 2025Image Editing As Programs with Diffusion Models
- 2025X2Edit: Revisiting Arbitrary-Instruction Image Editing through Self-Constructed Data and Task-Aware Representation Learning
- 2026Instruction-based Image Editing with Planning, Reasoning, and Generation