# Unsupervised Joint Mining of Deep Features and Image Labels for Large-scale Radiology Image Categorization and Scene Recognition

arXiv: https://arxiv.org/abs/1701.06599 · HTML: https://mlsanity.com/paper/1701.06599/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 23 Jan 2017.

## Built on by (most cited first, 5 of 5)

- [Combining pretrained CNN feature extractors to enhance clustering of complex natural images](https://mlsanity.com/paper/2101.02767/) (2021)
- [DeepMCAT: Large-Scale Deep Clustering for Medical Image Categorization](https://mlsanity.com/paper/2110.00109/) (2021)
- [Fisher Information based Stochastic Gradient Ascent for Online Learning of Dirichlet Process Mixture and Theory](https://mlsanity.com/paper/2412.08951/) (2024)
- [Identifying Signatures of Image Phenotypes to Track Treatment Response in Liver Disease](https://mlsanity.com/paper/2507.12012/) (2025)
- [PYPM-GGD: Pitman-Yor Process Mixture with Generalized Gaussian Density using ADAM](https://mlsanity.com/paper/2607.24583/) (2026)

## Abstract (arXiv)

The recent rapid and tremendous success of deep convolutional neural networks (CNN) on many challenging computer vision tasks largely derives from the accessibility of the well-annotated ImageNet and PASCAL VOC datasets. Nevertheless, unsupervised image categorization (i.e., without the ground-truth labeling) is much less investigated, yet critically important and difficult when annotations are extremely hard to obtain in the conventional way of "Google Search" and crowd sourcing. We address this problem by presenting a looped deep pseudo-task optimization (LDPO) framework for joint mining of deep CNN features and image labels. Our method is conceptually simple and rests upon the hypothesized "convergence" of better labels leading to better trained CNN models which in turn feed more discriminative image representations to facilitate more meaningful clusters/labels. Our proposed method is validated in tackling two important applications: 1) Large-scale medical image annotation has always been a prohibitively expensive and easily-biased task even for well-trained radiologists. Significantly better image categorization results are achieved via our proposed approach compared to the previous state-of-the-art method. 2) Unsupervised scene recognition on representative and publicly available datasets with our proposed technique is examined. The LDPO achieves excellent quantitative scene classification results. On the MIT indoor scene dataset, it attains a clustering accuracy of 75.3%, compared to the state-of-the-art supervised classification accuracy of 81.0% (when both are based on the VGG-VD model).