mlsanityData to 14 Oct 2026

Organizing Unstructured Image Collections using Natural Language

arXiv 2410.05217 ↗First posted 7 Oct 2024cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce and study the novel task of Open-ended Semantic Multiple Clustering (OpenSMC). Given a large, unstructured image collection, the goal is to automatically discover several, diverse semantic clustering criteria (e.g., Activity or Location) from the images, and subsequently organize them according to the discovered criteria, without requiring any human input. Our framework, X-Cluster: eXploratory Clustering, treats text as a reasoning proxy: it concurrently scans the entire image collection, proposes candidate criteria in natural language, and groups images into meaningful clusters per criterion. This radically differs from previous works, which either assume predefined clustering criteria or fixed cluster counts. To evaluate X-Cluster, we create two new benchmarks, COCO-4C and Food-4C, each annotated with four distinct grouping criteria and corresponding cluster labels. Experiments show that X-Cluster can effectively reveal meaningful partitions on several datasets. Finally, we use X-Cluster to achieve various real-world applications, including uncovering hidden biases in text-to-image (T2I) generative models and analyzing image virality on social media. Project page: https://oatmealliu.github.io/xcluster.html

Key facts. Cited by 7 papers in the map (data through 14 Oct 2026). Builds on 88 of its 147 references that are in the map. First posted on arXiv 7 Oct 2024.

Builds on

88 of 147 references.

References · 88

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2023GPT-4 Technical Reportcited by 18,396
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  6. 2022Training language models to follow instructions with human feedbackcited by 15,971
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  9. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  10. 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
  11. 2023Visual Instruction Tuningcited by 7,869
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
38 more
  1. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  3. 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,356
  4. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  5. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2023Mistral 7Bcited by 3,164
  8. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  9. 2020Shortcut Learning in Deep Neural Networkscited by 1,969
  10. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  11. 2019Distributionally Robust Neural Networks for Group Shifts: On the Importance of Regularization for Worst-Case Generalizationcited by 1,247
  12. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,089
  13. 2016ConceptNet 5.5: An Open Multilingual Graph of General Knowledgecited by 1,067
  14. 2016An overview of gradient descent optimization algorithmscited by 1,033
  15. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,028
  16. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 767
  17. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Modelscited by 584
  18. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 523
  19. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 431
  20. 2024A Survey on Hallucination in Large Vision-Language Modelscited by 357
  21. 2020SCAN: Learning to Classify Images without Labelscited by 352
  22. 2024VideoAgent: Long-form Video Understanding with Large Language Model as Agentcited by 332
  23. 2022Easily Accessible Text-to-Image Generation Amplifies Demographic Stereotypes at Large Scalecited by 262
  24. 2022Generalized Category Discoverycited by 211
  25. 2023Scalable 3D Captioning with Pretrained Modelscited by 203
  26. 2023"Kelly is a Warm Person, Joseph is a Role Model": Gender Biases in LLM-Generated Reference Letterscited by 199
  27. 2021Large Pre-trained Language Models Contain Human-like Biases of What is Right and Wrong to Docited by 178
  28. 2023LLM Lies: Hallucinations are not Bugs, but Features as Adversarial Examplescited by 177
  29. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 172
  30. 2021Bias Out-of-the-Box: An Empirical Analysis of Intersectional Occupational Biases in Popular Generative Language Modelscited by 141
  31. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  32. 2023Social Biases through the Text-to-Image Generation Lenscited by 127
  33. 2023Change is Hard: A Closer Look at Subpopulation Shiftcited by 120
  34. 2021AutoNovel: Automatically Discovering and Learning Novel Visual Categoriescited by 116
  35. 2014Part-based R-CNNs for Fine-grained Category Detectioncited by 109
  36. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 107
  37. 2023Vision-by-Language for Training-Free Compositional Image Retrievalcited by 103
  38. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 2.

7 papers.

  1. 2025Open Ad-hoc Categorization with Contextualized Feature Learningcited by 3
  2. 2025Discovering Divergent Representations between Text-to-Image Modelscited by 2
  3. 2025Visual Chronicles: Using Multimodal LLMs to Analyze Massive Collections of Imagescited by 1
  4. 2025Deep Clustering via Gradual Community Detectioncited by 0
  5. 2025Interpretable Text-Guided Image Clustering via Iterative Searchcited by 0
  6. 2026Universal Guideline-Driven Image Clustering via a Hybrid LLM Agentcited by 0
  7. 2026Controlling Embedding Spaces with Text-Conditioned Transformationscited by 0