# Accuracy on In-Domain Samples Matters When Building Out-of-Domain detectors: A Reply to Marek et al. (2021)

arXiv: https://arxiv.org/abs/2205.11887 · HTML: https://mlsanity.com/paper/2205.11887/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 29 references that are in the map.
- First posted on arXiv 24 May 2022.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Deep Anomaly Detection with Outlier Exposure](https://mlsanity.com/paper/1812.04606/) (2018)
- [Training Confidence-calibrated Classifiers for Detecting Out-of-Distribution Samples](https://mlsanity.com/paper/1711.09325/) (2017)
- [An Evaluation Dataset for Intent Classification and Out-of-Scope Prediction](https://mlsanity.com/paper/1909.02027/) (2019)
- [SeqGAN: Sequence Generative Adversarial Nets with Policy Gradient](https://mlsanity.com/paper/1609.05473/) (2016)
- [Pretrained Transformers Improve Out-of-Distribution Robustness](https://mlsanity.com/paper/2004.06100/) (2020)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [A Large-Scale Chinese Short-Text Conversation Dataset](https://mlsanity.com/paper/2008.03946/) (2020)
- [Personalized Dialogue Generation with Diversified Traits](https://mlsanity.com/paper/1901.09672/) (2019)
- [A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Data](https://mlsanity.com/paper/1911.04700/) (2019)
- [Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detection](https://mlsanity.com/paper/2101.03778/) (2021)
- [Out-of-domain Detection for Natural Language Understanding in Dialog Systems](https://mlsanity.com/paper/1909.03862/) (2019)
- [EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Training](https://mlsanity.com/paper/2108.01547/) (2021)
- [MMChat: Multi-Modal Chat Dataset on Social Media](https://mlsanity.com/paper/2108.07154/) (2021)
- [Diversifying Dialog Generation via Adaptive Label Smoothing](https://mlsanity.com/paper/2105.14556/) (2021)
- [Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Data](https://mlsanity.com/paper/2009.09427/) (2020)
- [DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2109.12599/) (2021)
- [Stylized Dialogue Response Generation Using Stylized Unpaired Texts](https://mlsanity.com/paper/2009.12719/) (2020)
- [OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation](https://mlsanity.com/paper/2104.02484/) (2021)
- [Transferable Persona-Grounded Dialogues via Grounded Minimal Edits](https://mlsanity.com/paper/2109.07713/) (2021)
- [Energy-based Unknown Intent Detection with Data Manipulation](https://mlsanity.com/paper/2107.12542/) (2021)

## Abstract (arXiv)

We have noticed that Marek et al. (2021) try to re-implement our paper Zheng et al. (2020a) in their work "OodGAN: Generative Adversarial Network for Out-of-Domain Data Generation". Our paper proposes a model to generate pseudo OOD samples that are akin to IN-Domain (IND) input utterances. These pseudo OOD samples can be used to improve the OOD detection performance by optimizing an entropy regularization term when building the IND classifier. Marek et al. (2021) report a large gap between their re-implemented results and ours on the CLINC150 dataset (Larson et al., 2019). This paper discusses some key observations that may have led to such a large gap. Most of these observations originate from our experiments because Marek et al. (2021) have not released their codes1. One of the most important observations is that stronger IND classifiers usually exhibit a more robust ability to detect OOD samples. We hope these observations help other researchers, including Marek et al. (2021), to develop better OOD detectors in their applications.