# Ask2Transformers: Zero-Shot Domain labelling with Pre-trained Language Models

arXiv: https://arxiv.org/abs/2101.02661 · HTML: https://mlsanity.com/paper/2101.02661/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 7 Jan 2021.

## Builds on (most cited first, 12 of 12)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [Unsupervised Cross-lingual Representation Learning at Scale](https://mlsanity.com/paper/1911.02116/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approach](https://mlsanity.com/paper/1909.00161/) (2019)
- [Denoising based Sequence-to-Sequence Pre-training for Text Generation](https://mlsanity.com/paper/1908.08206/) (2019)

## Built on by (most cited first, 13 of 13)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [Label Verbalization and Entailment for Effective Zero- and Few-Shot Relation Extraction](https://mlsanity.com/paper/2109.03659/) (2021)
- [Language Model as an Annotator: Exploring DialoGPT for Dialogue Summarization](https://mlsanity.com/paper/2105.12544/) (2021)
- [Zero-Shot Text Classification with Self-Training](https://mlsanity.com/paper/2210.17541/) (2022)
- [Textual Entailment for Event Argument Extraction: Zero- and Few-Shot with Multi-Source Learning](https://mlsanity.com/paper/2205.01376/) (2022)
- [What do Language Models know about word senses? Zero-Shot WSD with Language Models and Domain Inventories](https://mlsanity.com/paper/2302.03353/) (2023)
- [Distilling Hypernymy Relations from Language Models: On the Effectiveness of Zero-Shot Taxonomy Induction](https://mlsanity.com/paper/2202.04876/) (2022)
- [ZS4IE: A toolkit for Zero-Shot Information Extraction with simple Verbalizations](https://mlsanity.com/paper/2203.13602/) (2022)
- [Template-based Approach to Zero-shot Intent Recognition](https://mlsanity.com/paper/2206.10914/) (2022)
- [Pointing out the Shortcomings of Relation Extraction Models with Semantically Motivated Adversarials](https://mlsanity.com/paper/2402.19076/) (2024)
- [Text2Traj2Text: Learning-by-Synthesis Framework for Contextual Captioning of Human Movement Trajectories](https://mlsanity.com/paper/2409.12670/) (2024)
- [Why Do Neural Language Models Still Need Commonsense Knowledge to Handle Semantic Variations in Question Answering?](https://mlsanity.com/paper/2209.00599/) (2022)
- [Medical Argument Mining: Exploitation of Scarce Data Using NLI Systems](https://mlsanity.com/paper/2506.12823/) (2025)

## Abstract (arXiv)

In this paper we present a system that exploits different pre-trained Language Models for assigning domain labels to WordNet synsets without any kind of supervision. Furthermore, the system is not restricted to use a particular set of domain labels. We exploit the knowledge encoded within different off-the-shelf pre-trained Language Models and task formulations to infer the domain label of a particular WordNet definition. The proposed zero-shot system achieves a new state-of-the-art on the English dataset used in the evaluation.