# Towards Real-world Scenario: Imbalanced New Intent Discovery

arXiv: https://arxiv.org/abs/2406.03127 · HTML: https://mlsanity.com/paper/2406.03127/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 41 references that are in the map.
- First posted on arXiv 5 Jun 2024.

## Builds on (most cited first, 29 of 29)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Improved Training of Wasserstein GANs](https://mlsanity.com/paper/1704.00028/) (2017)
- [Class-Balanced Loss Based on Effective Number of Samples](https://mlsanity.com/paper/1901.05555/) (2019)
- [Deep Clustering for Unsupervised Learning of Visual Features](https://mlsanity.com/paper/1807.05520/) (2018)
- [Prototypical Contrastive Learning of Unsupervised Representations](https://mlsanity.com/paper/2005.04966/) (2020)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Efficient Intent Detection with Dual Sentence Encoders](https://mlsanity.com/paper/2003.04807/) (2020)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [Discovering New Intents with Deep Aligned Clustering](https://mlsanity.com/paper/2012.08987/) (2020)
- [Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinement](https://mlsanity.com/paper/1911.08891/) (2019)
- [Generalized Category Discovery with Decoupled Prototypical Network](https://mlsanity.com/paper/2211.15115/) (2022)
- [Multilingual Machine Translation Systems from Microsoft for WMT21 Shared Task](https://mlsanity.com/paper/2111.02086/) (2021)
- [Sinkhorn Label Allocation: Semi-Supervised Classification via Annealed Self-Training](https://mlsanity.com/paper/2102.08622/) (2021)
- [Generalized Zero-shot Intent Detection via Commonsense Knowledge](https://mlsanity.com/paper/2102.02925/) (2021)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [Novel Class Discovery for Long-tailed Recognition](https://mlsanity.com/paper/2308.02989/) (2023)
- [CROP: Zero-shot Cross-lingual Named Entity Recognition with Multilingual Labeled Sequence Translation](https://mlsanity.com/paper/2210.07022/) (2022)
- [mCL-NER: Cross-Lingual Named Entity Recognition via Multi-view Contrastive Learning](https://mlsanity.com/paper/2308.09073/) (2023)
- [Generalized Intent Discovery: Learning from Open World Dialogue System](https://mlsanity.com/paper/2209.06030/) (2022)
- [A Diffusion Weighted Graph Framework for New Intent Discovery](https://mlsanity.com/paper/2310.15836/) (2023)
- [New Intent Discovery with Attracting and Dispersing Prototype](https://mlsanity.com/paper/2403.16913/) (2024)
- [m3P: Towards Multimodal Multilingual Translation with Multimodal Prompt](https://mlsanity.com/paper/2403.17556/) (2024)
- [ImbaGCD: Imbalanced Generalized Category Discovery](https://mlsanity.com/paper/2401.05353/) (2023)
- [Going beyond research datasets: Novel intent discovery in the industry setting](https://mlsanity.com/paper/2305.05474/) (2023)
- [RoNID: New Intent Discovery with Generated-Reliable Labels and Cluster-friendly Representations](https://mlsanity.com/paper/2404.08977/) (2024)

## Built on by (most cited first, 2 of 2)

- [H2HTalk: Evaluating Large Language Models as Emotional Companion](https://mlsanity.com/paper/2507.03543/) (2025)
- [SWI: Speaking with Intent in Large Language Models](https://mlsanity.com/paper/2503.21544/) (2025)

## Abstract (arXiv)

New Intent Discovery (NID) aims at detecting known and previously undefined categories of user intent by utilizing limited labeled and massive unlabeled data. Most prior works often operate under the unrealistic assumption that the distribution of both familiar and new intent classes is uniform, overlooking the skewed and long-tailed distributions frequently encountered in real-world scenarios. To bridge the gap, our work introduces the imbalanced new intent discovery (i-NID) task, which seeks to identify familiar and novel intent categories within long-tailed distributions. A new benchmark (ImbaNID-Bench) comprised of three datasets is created to simulate the real-world long-tail distributions. ImbaNID-Bench ranges from broad cross-domain to specific single-domain intent categories, providing a thorough representation of practical use cases. Besides, a robust baseline model ImbaNID is proposed to achieve cluster-friendly intent representations. It includes three stages: model pre-training, generation of reliable pseudo-labels, and robust representation learning that strengthens the model performance to handle the intricacies of real-world data distributions. Our extensive experiments on previous benchmarks and the newly established benchmark demonstrate the superior performance of ImbaNID in addressing the i-NID task, highlighting its potential as a powerful baseline for uncovering and categorizing user intents in imbalanced and long-tailed distributionsfootnoteurlhttps://github.com/Zkdc/i-NID.