mlsanityData to 14 Oct 2026

MosaicFusion: Diffusion Models as Data Augmenters for Large Vocabulary Instance Segmentation

arXiv 2309.13042 ↗First posted 22 Sep 2023cs.CV cs.AI cs.LGcited by 35

On the map: Diffusion models → Medical imaging and segmentation

Abstract (arXiv)

We present MosaicFusion, a simple yet effective diffusion-based data augmentation approach for large vocabulary instance segmentation. Our method is training-free and does not rely on any label supervision. Two key designs enable us to employ an off-the-shelf text-to-image diffusion model as a useful dataset generator for object instances and mask annotations. First, we divide an image canvas into several regions and perform a single round of diffusion process to generate multiple instances simultaneously, conditioning on different text prompts. Second, we obtain corresponding instance masks by aggregating cross-attention maps associated with object prompts across layers and diffusion time steps, followed by simple thresholding and edge-aware refinement processing. Without bells and whistles, our MosaicFusion can produce a significant amount of synthetic labeled data for both rare and novel categories. Experimental results on the challenging LVIS long-tailed and open-vocabulary benchmarks demonstrate that MosaicFusion can significantly improve the performance of existing instance segmentation models, especially for rare and novel categories. Code: https://github.com/Jiahao000/MosaicFusion.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Medical imaging and segmentation, one of 14 directions in Diffusion models. Builds on 79 of its 105 references that are in the map. First posted on arXiv 22 Sep 2023.

Builds on

79 of 105 references.

Closest ancestors · 1

Same direction, newest first

  1. 2021Label-Efficient Semantic Segmentation with Diffusion Models6 Dec

Other references · 78

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  8. 2017Mask R-CNNcited by 7,277
  9. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  11. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  12. 2016Feature Pyramid Networks for Object Detectioncited by 5,036
38 more
  1. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  2. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  3. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  4. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  5. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  6. 2020YOLOv4: Optimal Speed and Accuracy of Object Detectioncited by 1,630
  7. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,433
  8. 2019MMDetection: Open MMLab Detection Toolbox and Benchmarkcited by 1,385
  9. 2021Palette: Image-to-Image Diffusion Modelscited by 1,300
  10. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,217
  11. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,106
  12. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,020
  13. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 906
  14. 2015Generation and Comprehension of Unambiguous Object Descriptionscited by 904
  15. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  16. 2016Playing for Data: Ground Truth from Computer Gamescited by 818
  17. 2022Language-driven Semantic Segmentationcited by 717
  18. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 702
  19. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  20. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  21. 2020Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentationcited by 596
  22. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  23. 2023Zero-shot Image-to-Image Translationcited by 505
  24. 2021Scaling Open-Vocabulary Image Segmentation with Image-Level Labelscited by 473
  25. 2020Open-Vocabulary Object Detection Using Captionscited by 427
  26. 2020Balanced Meta-Softmax for Long-Tailed Visual Recognitioncited by 400
  27. 2022Learning to Prompt for Open-Vocabulary Object Detection with Vision-Language Modelcited by 317
  28. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  29. 2020Equalization Loss for Long-Tailed Object Recognitioncited by 285
  30. 2023BoxDiff: Text-to-Image Synthesis with Training-Free Box-Constrained Diffusioncited by 282
  31. 2022DETRs with Collaborative Hybrid Assignments Trainingcited by 274
  32. 2021DatasetGAN: Efficient Labeled Data Factory with Minimal Human Effortcited by 264
  33. 2022Open-Vocabulary DETR with Conditional Matchingcited by 220
  34. 2017Cut, Paste and Learn: Surprisingly Easy Synthesis for Instance Detectioncited by 209
  35. 2023Encoder-based Domain Tuning for Fast Personalization of Text-to-Image Modelscited by 203
  36. 2021Distribution Alignment: A Unified Framework for Long-tail Visual Recognitioncited by 198
  37. 2019iSAID: A Large-scale Dataset for Instance Segmentation in Aerial Imagescited by 197
  38. 2023Towards Open Vocabulary Learning: A Surveycited by 166

50 most cited of 78 All 78 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 5; 2024: 14; 2025: 11; 2026: 5.

35 papers.

Sort:

By area:

  1. 2024MotionBooth: Motion-Aware Customized Text-to-Video Generationcited by 93
  2. 2024Open-Vocabulary SAM: Segment and Recognize Twenty-thousand Classes Interactivelycited by 75
  3. 2024Image Segmentation in Foundation Model Era: A Surveycited by 35
  4. 2024SemFlow: Binding Semantic Segmentation and Image Synthesis via Rectified Flowcited by 20
  5. 2023SegGen: Supercharging Segmentation Models with Text2Mask and Mask2Img Synthesiscited by 18
  6. 2024DiverGen: Improving Instance Segmentation by Learning Wider Data Distribution with More Diverse Generative Datacited by 15
  7. 2023AI-Generated Images as Data Source: The Dawn of Synthetic Eracited by 13
  8. 2024Explore In-Context Segmentation via Latent Diffusion Modelscited by 12
  9. 2024Generative Active Learning for Long-tailed Instance Segmentationcited by 10
  10. 2025A Comprehensive Survey on Imbalanced Data Learningcited by 10
  11. 2024GenView: Enhancing View Quality with Pretrained Generative Model for Self-Supervised Learningcited by 9
  12. 2023OV-VG: A Benchmark for Open-Vocabulary Visual Groundingcited by 7
23 more
  1. 2025ReCon: Region-Controllable Data Augmentation with Rectification and Alignment for Object Detectioncited by 6
  2. 2023ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentationcited by 5
  3. 2024WAS: Dataset and Methods for Artistic Text Segmentationcited by 5
  4. 2024DODA: Adapting Object Detectors to Dynamic Agricultural Environments in Real-Time with Diffusioncited by 4
  5. 2024RTGen: Generating Region-Text Pairs for Open-Vocabulary Object Detectioncited by 4
  6. 2024Auto Cherry-Picker: Learning from High-quality Generative Data Driven by Languagecited by 4
  7. 2025TRACE: Your Diffusion Model is Secretly an Instance Edge Detectorcited by 4
  8. 2024Weak-to-Strong Compositional Learning from Generative Models for Language-based Object Detectioncited by 3
  9. 2025Gen-n-Val: Agentic Image Data Generation and Validationcited by 3
  10. 2025DreamMask: Boosting Open-vocabulary Panoptic Segmentation with Synthetic Datacited by 2
  11. 2025DIVE: Inverting Conditional Diffusion Models for Discriminative Taskscited by 2
  12. 2026On the Difficulty of Learning a Meta-network for Training Data Selectioncited by 2
  13. 2023DST-Det: Simple Dynamic Self-Training for Open-Vocabulary Object Detectioncited by 1
  14. 2024Paint Outside the Box: Synthesizing and Selecting Training Data for Visual Groundingcited by 1
  15. 2025Scene-Aware Location Modeling for Data Augmentation in Automotive Object Detectioncited by 1
  16. 2025SynRES: Towards Referring Expression Segmentation in the Wild via Synthetic Datacited by 1
  17. 2025Geometry-Editable and Appearance-Preserving Object Compositoncited by 1
  18. 2026Leveraging Image Generators to Address Data Scarcity: The Gen4Regen Dataset for Forest Regeneration Mappingcited by 1
  19. 2026TMI: Text-to-Image Meets Image-to-Image for Complementary Data Synthesis to Boost Long-Tailed Instance Segmentationcited by 1
  20. 2025Prompt-Free Conditional Diffusion for Multi-object Image Augmentationcited by 0
  21. 2025Can Synthetic Images Serve as Effective and Efficient Class Prototypes?cited by 0
  22. 2026What Makes Synthetic Data Effective in Image Segmentationcited by 0
  23. 2026Representation Learning in Diffusion and Flow-based Model: An Application Aspectcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Open-vocabulary Object Segmentation with Diffusion Models
  2. 2023Diffusion Models for Open-Vocabulary Segmentation
  3. 2026Diffuse2Seg: Diffusion Models Can Segment Anything Without Supervision
  4. 2023Open-Vocabulary Panoptic Segmentation with Text-to-Image Diffusion Models
  5. 2024FreeSeg-Diff: Training-Free Open-Vocabulary Segmentation with Diffusion Models
  6. 2023Catch Me If You Can Describe Me: Open-Vocabulary Camouflaged Instance Segmentation with Diffusion
  7. 2026From Diffusion to Rectified Flow: Rethinking Text-Based Segmentation
  8. 2025FA-Seg: A Fast and Accurate Diffusion-Based Method for Open-Vocabulary Segmentation