mlsanityData to 7 Oct 2026

Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data

arXiv 2104.12673 ↗First posted 26 Apr 2021cs.CVcited by 55

Not on the map. Why?

Abstract (arXiv)

This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further employ Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.

Key facts. Cited by 55 papers in the map (data through 7 Oct 2026). Builds on 37 of its 55 references that are in the map. First posted on arXiv 26 Apr 2021.

Builds on

37 of 55 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2014Going Deeper with Convolutionscited by 4,231
  6. 2020Supervised Contrastive Learningcited by 2,803
  7. 2017The Kinetics Human Action Video Datasetcited by 1,894
  8. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  9. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  10. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2019Objects as Pointscited by 1,172
25 more
  1. 2017A Closer Look at Spatiotemporal Convolutions for Action Recognitioncited by 1,139
  2. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  3. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  4. 2019EfficientDet: Scalable and Efficient Object Detectioncited by 1,018
  5. 2015Unsupervised Deep Embedding for Clustering Analysiscited by 763
  6. 2020VGGSound: A Large-scale Audio-Visual Datasetcited by 641
  7. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 542
  8. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  9. 2018Invariant Information Clustering for Unsupervised Image Classification and Segmentationcited by 454
  10. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  11. 2019Self-Supervised Learning by Cross-Modal Audio-Video Clusteringcited by 264
  12. 2019Learning to Discover Novel Visual Categories via Deep Transfer Clusteringcited by 223
  13. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  14. 2020Audio-Visual Instance Discrimination with Cross-Modal Agreementcited by 192
  15. 2020Automatically Discovering and Learning New Visual Categories with Ranking Statisticscited by 141
  16. 2017Learning to cluster in order to transfer across domains and taskscited by 128
  17. 2017Deep Clustering via Joint Convolutional Autoencoder Embedding and Relative Entropy Minimizationcited by 113
  18. 2019Multi-class Classification without Multi-class Labelscited by 100
  19. 2020Labelling unlabelled videos from scratch with multi-modal self-supervisioncited by 99
  20. 2020OpenMix: Reviving Known Knowledge for Discovering Novel Visual Categories in An Open Worldcited by 93
  21. 2019Semi-Supervised Learning with Scarce Annotationscited by 25
  22. 2020Look-into-Object: Self-supervised Structure Modeling for Object Recognitioncited by 20
  23. 2020LSD-C: Linearly Separable Deep Clusterscited by 16
  24. 2020Boosting Deep Open World Recognition by Clusteringcited by 9
  25. 2018A probabilistic constrained clustering for transfer learning and image category discoverycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 12; 2023: 13; 2024: 12; 2025: 9; 2026: 6.

55 papers.

Sort:

By area:

  1. 2021Generalized Out-of-Distribution Detection: A Surveycited by 796
  2. 2022Generalized Category Discoverycited by 209
  3. 2021A Unified Objective for Novel Class Discoverycited by 161
  4. 2021Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillationcited by 104
  5. 2022PromptCAL: Contrastive Affinity Learning via Auxiliary Prompts for Generalized Novel Category Discoverycited by 85
  6. 2023Learning Semi-supervised Gaussian Mixture Models for Generalized Category Discoverycited by 72
  7. 2022Towards Open-Set Object Detection and Discoverycited by 44
  8. 2022Class-incremental Novel Class Discoverycited by 42
  9. 2023MetaGCD: Learning to Continually Learn in Generalized Category Discoverycited by 40
  10. 2022Novel Class Discovery without Forgettingcited by 39
  11. 2022Parametric Information Maximization for Generalized Category Discoverycited by 39
  12. 2022OpenLDN: Learning to Discover Novel Classes for Open-World Semi-Supervised Learningcited by 37
38 more
  1. 2024Generalized Out-of-Distribution Detection and Beyond in Vision Language Model Era: A Surveycited by 36
  2. 2022Grow and Merge: A Unified Framework for Continuous Categories Discoverycited by 33
  3. 2022Modeling Inter-Class and Intra-Class Constraints in Novel Class Discoverycited by 26
  4. 2023CiPR: An Efficient Framework with Cross-instance Positive Relations for Generalized Category Discoverycited by 26
  5. 2023Novel Class Discovery: an Introduction and Key Conceptscited by 24
  6. 2024HiLo: A Learning Framework for Generalized Category Discovery Robust to Domain Shiftscited by 22
  7. 2023Novel Class Discovery for 3D Point Cloud Semantic Segmentationcited by 21
  8. 2023When and How Does Known Class Help Discover Unknown Ones? Provable Understanding Through Spectral Analysiscited by 21
  9. 2025Hyperbolic Category Discoverycited by 20
  10. 2024Prototypical Hash Encoding for On-the-Fly Fine-Grained Category Discoverycited by 19
  11. 2023Proxy Anchor-based Unsupervised Learning for Continuous Generalized Category Discoverycited by 16
  12. 2024Effective Prompt Pool Learning for Continual Category Discoverycited by 14
  13. 2025DebGCD: Debiased Learning with Distribution Guidance for Generalized Category Discoverycited by 14
  14. 2024Novel Class Discovery for Ultra-Fine-Grained Visual Categorizationcited by 13
  15. 2022Learning to Discover and Detect Objectscited by 11
  16. 2023Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discoverycited by 11
  17. 2022Spacing Loss for Discovering Novel Categoriescited by 10
  18. 2023What's in a Name? Beyond Class Indices for Image Recognitioncited by 9
  19. 2025Generate, Refine, and Encode: Leveraging Synthesized Novel Samples for On-the-Fly Fine-Grained Category Discoverycited by 9
  20. 2024Multimodal Generalized Category Discoverycited by 7
  21. 2025SEAL: Semantic-Aware Hierarchical Learning for Generalized Category Discoverycited by 7
  22. 2024Adaptive Discovering and Merging for Incremental Novel Class Discoverycited by 6
  23. 2022Automatically Discovering Novel Visual Categories with Self-supervised Prototype Learningcited by 5
  24. 2024Online Continuous Generalized Category Discoverycited by 5
  25. 2025Category Discovery: An Open-World Perspectivecited by 5
  26. 2023Novel class discovery meets foundation models for 3D semantic segmentationcited by 4
  27. 2024Beyond Known Clusters: Probe New Prototypes for Efficient Generalized Class Discoverycited by 4
  28. 2023Novel Categories Discovery Via Constraints on Empirical Prediction Statisticscited by 3
  29. 2023NEV-NCD: Negative Learning, Entropy, and Variance regularization based novel action categories discoverycited by 2
  30. 2024PANDAS: Prototype-based Novel Class Discovery and Detectioncited by 2
  31. 2024Self-Cooperation Knowledge Distillation for Novel Class Discoverycited by 2
  32. 2025Novel Category Discovery with X-Agent Attention for Open-Vocabulary Semantic Segmentationcited by 2
  33. 2025PartCo: Part-Level Correspondence Priors Enhance Category Discoverycited by 2
  34. 2023A Critical Look at the Current Usage of Foundation Model for Dense Recognition Taskcited by 1
  35. 2025Generalized Category Discovery under the Long-Tailed Distributioncited by 1
  36. 2026TALON: Test-time Adaptive Learning for On-the-Fly Category Discoverycited by 1
  37. 2024Unseen Object Reasoning with Shared Appearance Cuescited by 0
  38. 2025Exploring the Tradeoff Between Diversity and Discrimination for Continuous Category Discoverycited by 0

50 most cited of 55 All 55 (JSON)