# Learning to cluster in order to transfer across domains and tasks

arXiv: https://arxiv.org/abs/1711.10125 · HTML: https://mlsanity.com/paper/1711.10125/

## Key facts

- Cited by 128 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 28 Nov 2017.

## Built on by (most cited first, 30 of 128)

- [FedDG: Federated Domain Generalization on Medical Image Segmentation via Episodic Learning in Continuous Frequency Space](https://mlsanity.com/paper/2103.06030/) (2021)
- [Generalized Category Discovery](https://mlsanity.com/paper/2201.02609/) (2022)
- [Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2102.03526/) (2021)
- [A Unified Objective for Novel Class Discovery](https://mlsanity.com/paper/2108.08536/) (2021)
- [Neighborhood Contrastive Learning for Novel Class Discovery](https://mlsanity.com/paper/2106.10731/) (2021)
- [AutoNovel: Automatically Discovering and Learning Novel Visual Categories](https://mlsanity.com/paper/2106.15252/) (2021)
- [Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation](https://mlsanity.com/paper/2107.03358/) (2021)
- [Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discovery](https://mlsanity.com/paper/2305.06144/) (2023)
- [Towards Discovery and Attribution of Open-world GAN Generated Images](https://mlsanity.com/paper/2105.04580/) (2021)
- [Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data](https://mlsanity.com/paper/2104.12673/) (2021)
- [XCon: Learning with Experts for Fine-grained Category Discovery](https://mlsanity.com/paper/2208.01898/) (2022)
- [Generalizable Cross-modality Medical Image Segmentation via Style Augmentation and Dual Normalization](https://mlsanity.com/paper/2112.11177/) (2021)
- [Class-incremental Novel Class Discovery](https://mlsanity.com/paper/2207.08605/) (2022)
- [FedDrive: Generalizing Federated Learning to Semantic Segmentation in Autonomous Driving](https://mlsanity.com/paper/2202.13670/) (2022)
- [UC-OWOD: Unknown-Classified Open World Object Detection](https://mlsanity.com/paper/2207.11455/) (2022)
- [Novel Class Discovery without Forgetting](https://mlsanity.com/paper/2207.10659/) (2022)
- [OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learning](https://mlsanity.com/paper/2207.02261/) (2022)
- [SelEx: Self-Expertise in Fine-Grained Generalized Category Discovery](https://mlsanity.com/paper/2408.14371/) (2024)
- [OpenCon: Open-world Contrastive Learning](https://mlsanity.com/paper/2208.02764/) (2022)
- [Meta Discovery: Learning to Discover Novel Classes given Very Limited Data](https://mlsanity.com/paper/2102.04002/) (2021)
- [Novel Class Discovery in Semantic Segmentation](https://mlsanity.com/paper/2112.01900/) (2021)
- [New Intent Discovery with Pre-training and Contrastive Learning](https://mlsanity.com/paper/2205.12914/) (2022)
- [Towards Realistic Semi-Supervised Learning](https://mlsanity.com/paper/2207.02269/) (2022)
- [Federated Generalized Category Discovery](https://mlsanity.com/paper/2305.14107/) (2023)
- [Contrastive Pseudo Learning for Open-World DeepFake Attribution](https://mlsanity.com/paper/2309.11132/) (2023)
- [GET: Unlocking the Multi-modal Potential of CLIP for Generalized Category Discovery](https://mlsanity.com/paper/2403.09974/) (2024)
- [Modeling Inter-Class and Intra-Class Constraints in Novel Class Discovery](https://mlsanity.com/paper/2210.03591/) (2022)
- [CiPR: An Efficient Framework with Cross-instance Positive Relations for Generalized Category Discovery](https://mlsanity.com/paper/2304.06928/) (2023)
- [Incremental Generalized Category Discovery](https://mlsanity.com/paper/2304.14310/) (2023)
- [CLIP-GCD: Simple Language Guided Generalized Category Discovery](https://mlsanity.com/paper/2305.10420/) (2023)

## Abstract (arXiv)

This paper introduces a novel method to perform transfer learning across domains and tasks, formulating it as a problem of learning to cluster. The key insight is that, in addition to features, we can transfer similarity information and this is sufficient to learn a similarity function and clustering network to perform both domain adaptation and cross-task transfer learning. We begin by reducing categorical information to pairwise constraints, which only considers whether two instances belong to the same class or not. This similarity is category-agnostic and can be learned from data in the source domain using a similarity network. We then present two novel approaches for performing transfer learning using this similarity function. First, for unsupervised domain adaptation, we design a new loss function to regularize classification with a constrained clustering loss, hence learning a clustering network with the transferred similarity metric generating the training inputs. Second, for cross-task learning (i.e., unsupervised clustering with unseen categories), we propose a framework to reconstruct and estimate the number of semantic clusters, again using the clustering network. Since the similarity network is noisy, the key is to use a robust clustering algorithm, and we show that our formulation is more robust than the alternative constrained and unconstrained clustering approaches. Using this method, we first show state of the art results for the challenging cross-task problem, applied on Omniglot and ImageNet. Our results show that we can reconstruct semantic clusters with high accuracy. We then evaluate the performance of cross-domain transfer using images from the Office-31 and SVHN-MNIST tasks and present top accuracy on both datasets. Our approach doesn't explicitly deal with domain discrepancy. If we combine with a domain adaptation loss, it shows further improvement.