# Estimating Semantic Similarity between In-Domain and Out-of-Domain Samples

arXiv: https://arxiv.org/abs/2306.01206 · HTML: https://mlsanity.com/paper/2306.01206/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 34 references that are in the map.
- First posted on arXiv 1 Jun 2023.

## Builds on (most cited first, 25 of 25)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Character-level Convolutional Networks for Text Classification](https://mlsanity.com/paper/1509.01626/) (2015)
- [CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledge](https://mlsanity.com/paper/1811.00937/) (2018)
- [Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shift](https://mlsanity.com/paper/1906.02530/) (2019)
- [Crowdsourcing Multiple Choice Science Questions](https://mlsanity.com/paper/1707.06209/) (2017)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [QASC: A Dataset for Question Answering via Sentence Composition](https://mlsanity.com/paper/1910.11473/) (2019)
- [MRQA 2019 Shared Task: Evaluating Generalization in Reading Comprehension](https://mlsanity.com/paper/1910.09753/) (2019)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [From GAN to WGAN](https://mlsanity.com/paper/1904.08994/) (2019)
- [Metaphors in Pre-Trained Language Models: Probing and Generalization Across Datasets and Languages](https://mlsanity.com/paper/2203.14139/) (2022)
- [Generalized but not Robust? Comparing the Effects of Data Modification Methods on Out-of-Domain Generalization and Adversarial Robustness](https://mlsanity.com/paper/2203.07653/) (2022)
- [On Continual Model Refinement in Out-of-Distribution Data Streams](https://mlsanity.com/paper/2205.02014/) (2022)
- [An Empirical Study on Explanations in Out-of-Domain Settings](https://mlsanity.com/paper/2203.00056/) (2022)
- [ILDAE: Instance-Level Difficulty Analysis of Evaluation Data](https://mlsanity.com/paper/2203.03073/) (2022)
- [Towards Textual Out-of-Domain Detection without In-Domain Labels](https://mlsanity.com/paper/2203.11396/) (2022)
- [Improving Generalizability in Implicitly Abusive Language Detection with Concept Activation Vectors](https://mlsanity.com/paper/2204.02261/) (2022)
- [The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error](https://mlsanity.com/paper/2203.11317/) (2022)
- [Fine-Tuning Deteriorates General Textual Out-of-Distribution Detection by Distorting Task-Agnostic Features](https://mlsanity.com/paper/2301.12715/) (2023)
- [Robust Natural Language Processing: Recent Advances, Challenges, and Future Directions](https://mlsanity.com/paper/2201.00768/) (2022)
- [Understanding Out-of-distribution: A Perspective of Data Dynamics](https://mlsanity.com/paper/2111.14730/) (2021)
- [Assessing Out-of-Domain Language Model Performance from Few Examples](https://mlsanity.com/paper/2210.06725/) (2022)
- [Self-Supervised Losses for One-Class Textual Anomaly Detection](https://mlsanity.com/paper/2204.05695/) (2022)
- [A Proposal to Study "Is High Quality Data All We Need?"](https://mlsanity.com/paper/2203.06404/) (2022)
- [Understanding, Detecting, and Separating Out-of-Distribution Samples and Adversarial Samples in Text Classification](https://mlsanity.com/paper/2204.04458/) (2022)

## Built on by (most cited first, 1 of 1)

- [Can Humans Identify Domains?](https://mlsanity.com/paper/2404.01785/) (2024)

## Abstract (arXiv)

Prior work typically describes out-of-domain (OOD) or out-of-distribution (OODist) samples as those that originate from dataset(s) or source(s) different from the training set but for the same task. When compared to in-domain (ID) samples, the models have been known to usually perform poorer on OOD samples, although this observation is not consistent. Another thread of research has focused on OOD detection, albeit mostly using supervised approaches. In this work, we first consolidate and present a systematic analysis of multiple definitions of OOD and OODist as discussed in prior literature. Then, we analyze the performance of a model under ID and OOD/OODist settings in a principled way. Finally, we seek to identify an unsupervised method for reliably identifying OOD/OODist samples without using a trained model. The results of our extensive evaluation using 12 datasets from 4 different tasks suggest the promising potential of unsupervised metrics in this task.