# Toward Open-domain Slot Filling via Self-supervised Co-training

arXiv: https://arxiv.org/abs/2303.13801 · HTML: https://mlsanity.com/paper/2303.13801/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 62 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [On the Variance of the Adaptive Learning Rate and Beyond](https://mlsanity.com/paper/1908.03265/) (2019)
- [Unsupervised Deep Embedding for Clustering Analysis](https://mlsanity.com/paper/1511.06335/) (2015)
- [Bidirectional LSTM-CRF Models for Sequence Tagging](https://mlsanity.com/paper/1508.01991/) (2015)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Few-shot Slot Tagging with Collapsed Dependency Transfer and Label-enhanced Task-adaptive Projection Network](https://mlsanity.com/paper/2006.05702/) (2020)
- [Joint Slot Filling and Intent Detection via Capsule Neural Networks](https://mlsanity.com/paper/1812.09471/) (2018)
- [DiSAN: Directional Self-Attention Network for RNN/CNN-Free Language Understanding](https://mlsanity.com/paper/1709.04696/) (2017)
- [Learning Named Entity Tagger using Domain-Specific Dictionary](https://mlsanity.com/paper/1809.03599/) (2018)
- [Few-shot classification in Named Entity Recognition Task](https://mlsanity.com/paper/1812.06158/) (2018)
- [A Sequence-to-Sequence Approach to Dialogue State Tracking](https://mlsanity.com/paper/2011.09553/) (2020)
- [Towards Zero-Shot Frame Semantic Parsing for Domain Scaling](https://mlsanity.com/paper/1707.02363/) (2017)
- [Robust Zero-Shot Cross-Domain Slot Filling with Example Values](https://mlsanity.com/paper/1906.06870/) (2019)
- [Coach: A Coarse-to-Fine Approach for Cross-domain Slot Filling](https://mlsanity.com/paper/2004.11727/) (2020)
- [Learning to Classify Intents and Slot Labels Given a Handful of Examples](https://mlsanity.com/paper/2004.10793/) (2020)
- [SwellShark: A Generative Model for Biomedical Named Entity Recognition without Labeled Data](https://mlsanity.com/paper/1704.06360/) (2017)
- [Leveraging Sentence-level Information with Encoder LSTM for Semantic Slot Filling](https://mlsanity.com/paper/1601.01530/) (2016)
- [Intent Mining from past conversations for conversational agent](https://mlsanity.com/paper/2005.11014/) (2020)
- [Deep Semantic Role Labeling with Self-Attention](https://mlsanity.com/paper/1712.01586/) (2017)
- [Generalized Zero-shot Intent Detection via Commonsense Knowledge](https://mlsanity.com/paper/2102.02925/) (2021)
- [Zero-Shot Adaptive Transfer for Conversational Language Understanding](https://mlsanity.com/paper/1808.10059/) (2018)
- [Optimal Hyperparameters for Deep LSTM-Networks for Sequence Labeling Tasks](https://mlsanity.com/paper/1707.06799/) (2017)

## Built on by (most cited first, 2 of 2)

- [Training Zero-Shot Generalizable End-to-End Task-Oriented Dialog System Without Turn-level Dialog Annotations](https://mlsanity.com/paper/2407.15055/) (2024)
- [Improving Multi-turn Task Completion in Task-Oriented Dialog Systems via Prompt Chaining and Fine-Grained Feedback](https://mlsanity.com/paper/2502.13298/) (2025)

## Abstract (arXiv)

Slot filling is one of the critical tasks in modern conversational systems. The majority of existing literature employs supervised learning methods, which require labeled training data for each new domain. Zero-shot learning and weak supervision approaches, among others, have shown promise as alternatives to manual labeling. Nonetheless, these learning paradigms are significantly inferior to supervised learning approaches in terms of performance. To minimize this performance gap and demonstrate the possibility of open-domain slot filling, we propose a Self-supervised Co-training framework, called SCot, that requires zero in-domain manually labeled training examples and works in three phases. Phase one acquires two sets of complementary pseudo labels automatically. Phase two leverages the power of the pre-trained language model BERT, by adapting it for the slot filling task using these sets of pseudo labels. In phase three, we introduce a self-supervised cotraining mechanism, where both models automatically select highconfidence soft labels to further improve the performance of the other in an iterative fashion. Our thorough evaluations show that SCot outperforms state-of-the-art models by 45.57% and 37.56% on SGD and MultiWoZ datasets, respectively. Moreover, our proposed framework SCot achieves comparable performance when compared to state-of-the-art fully supervised models.