# Span-ConveRT: Few-shot Span Extraction for Dialog with Pretrained Conversational Representations

arXiv: https://arxiv.org/abs/2005.08866 · HTML: https://mlsanity.com/paper/2005.08866/

## Key facts

- Cited by 42 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2020.

## Built on by (most cited first, 30 of 42)

- [Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Survey](https://mlsanity.com/paper/2111.01243/) (2021)
- [A Survey on Recent Advances in LLM-Based Multi-turn Dialogue Systems](https://mlsanity.com/paper/2402.18013/) (2024)
- [Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Survey](https://mlsanity.com/paper/2105.04387/) (2021)
- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [SPACE-3: Unified Dialog Model Pre-training for Task-Oriented Dialog Understanding and Generation](https://mlsanity.com/paper/2209.06664/) (2022)
- [Learning to Execute Actions or Ask Clarification Questions](https://mlsanity.com/paper/2204.08373/) (2022)
- [SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding](https://mlsanity.com/paper/2209.06638/) (2022)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [DialogStudio: Towards Richest and Most Diverse Unified Dataset Collection for Conversational AI](https://mlsanity.com/paper/2307.10172/) (2023)
- [Empowering the Fact-checkers! Automatic Identification of Claim Spans on Twitter](https://mlsanity.com/paper/2210.04710/) (2022)
- [Label Semantic Aware Pre-training for Few-shot Text Classification](https://mlsanity.com/paper/2204.07128/) (2022)
- [LAD: Language Models as Data for Zero-Shot Dialog](https://mlsanity.com/paper/2207.14393/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [A Survey of Intent Classification and Slot-Filling Datasets for Task-Oriented Dialog](https://mlsanity.com/paper/2207.13211/) (2022)
- [GenSF: Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling](https://mlsanity.com/paper/2106.07055/) (2021)
- [Generative Conversational Networks](https://mlsanity.com/paper/2106.08484/) (2021)
- [Improved and Efficient Conversational Slot Labeling through Question Answering](https://mlsanity.com/paper/2204.02123/) (2022)
- [UniDU: Towards A Unified Generative Dialogue Understanding Framework](https://mlsanity.com/paper/2204.04637/) (2022)
- [Prompt Conditioned VAE: Enhancing Generative Replay for Lifelong Learning in Task-Oriented Dialogue](https://mlsanity.com/paper/2210.07783/) (2022)
- [NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2204.13021/) (2022)
- [Prompter: Zero-shot Adaptive Prefixes for Dialogue State Tracking Domain Adaptation](https://mlsanity.com/paper/2306.04724/) (2023)
- [N-Shot Learning for Augmenting Task-Oriented Dialogue State Tracking](https://mlsanity.com/paper/2103.00293/) (2021)
- [Natural Language Understanding for Argumentative Dialogue Systems in the Opinion Building Domain](https://mlsanity.com/paper/2103.02691/) (2021)
- [A Survey of Ontology Expansion for Conversational Understanding](https://mlsanity.com/paper/2410.15019/) (2024)
- [Inverse is Better! Fast and Accurate Prompt for Few-shot Slot Tagging](https://mlsanity.com/paper/2204.00885/) (2022)
- [Transfer-Free Data-Efficient Multilingual Slot Labeling](https://mlsanity.com/paper/2305.13528/) (2023)
- [Overview of the CLAIMSCAN-2023: Uncovering Truth in Social Media through Claim Detection and Identification of Claim Spans](https://mlsanity.com/paper/2310.19267/) (2023)
- [SYNERGY: Building Task Bots at Scale Using Symbolic Knowledge and Machine Teaching](https://mlsanity.com/paper/2110.11514/) (2021)
- [DFM: Dialogue Foundation Model for Universal Large-Scale Dialogue-Oriented Task Learning](https://mlsanity.com/paper/2205.12662/) (2022)
- [Semantic-based Pre-training for Dialogue Understanding](https://mlsanity.com/paper/2209.09146/) (2022)

## Abstract (arXiv)

We introduce Span-ConveRT, a light-weight model for dialog slot-filling which frames the task as a turn-based span extraction task. This formulation allows for a simple integration of conversational knowledge coded in large pretrained conversational models such as ConveRT (Henderson et al., 2019). We show that leveraging such knowledge in Span-ConveRT is especially useful for few-shot learning scenarios: we report consistent gains over 1) a span extractor that trains representations from scratch in the target domain, and 2) a BERT-based span extractor. In order to inspire more work on span extraction for the slot-filling task, we also release RESTAURANTS-8K, a new challenging data set of 8,198 utterances, compiled from actual conversations in the restaurant booking domain.