# ConVEx: Data-Efficient and Few-Shot Slot Labeling

arXiv: https://arxiv.org/abs/2010.11791 · HTML: https://mlsanity.com/paper/2010.11791/

## Key facts

- Cited by 22 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 22 Oct 2020.

## Built on by (most cited first, 22 of 22)

- [InstructDial: Improving Zero and Few-shot Generalization in Dialogue through Instruction Tuning](https://mlsanity.com/paper/2205.12673/) (2022)
- [Controllable Semantic Parsing via Retrieval Augmentation](https://mlsanity.com/paper/2110.08458/) (2021)
- [SPACE-2: Tree-Structured Semi-Supervised Contrastive Pre-training for Task-Oriented Dialog Understanding](https://mlsanity.com/paper/2209.06638/) (2022)
- [LightNER: A Lightweight Tuning Paradigm for Low-resource NER via Pluggable Prompting](https://mlsanity.com/paper/2109.00720/) (2021)
- [An Enhanced Span-based Decomposition Method for Few-Shot Sequence Labeling](https://mlsanity.com/paper/2109.13023/) (2021)
- [ConvFiT: Conversational Fine-Tuning of Pretrained Language Models](https://mlsanity.com/paper/2109.10126/) (2021)
- [LAD: Language Models as Data for Zero-Shot Dialog](https://mlsanity.com/paper/2207.14393/) (2022)
- [Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systems](https://mlsanity.com/paper/2104.08570/) (2021)
- [Robust Dialogue State Tracking with Weak Supervision and Sparse Data](https://mlsanity.com/paper/2202.03354/) (2022)
- [GenSF: Simultaneous Adaptation of Generative Pre-trained Models and Slot Filling](https://mlsanity.com/paper/2106.07055/) (2021)
- [Unsupervised Slot Schema Induction for Task-oriented Dialog](https://mlsanity.com/paper/2205.04515/) (2022)
- [Improved and Efficient Conversational Slot Labeling through Question Answering](https://mlsanity.com/paper/2204.02123/) (2022)
- [NLU++: A Multi-Label, Slot-Rich, Generalisable Dataset for Natural Language Understanding in Task-Oriented Dialogue](https://mlsanity.com/paper/2204.13021/) (2022)
- [Inverse is Better! Fast and Accurate Prompt for Few-shot Slot Tagging](https://mlsanity.com/paper/2204.00885/) (2022)
- [Transfer-Free Data-Efficient Multilingual Slot Labeling](https://mlsanity.com/paper/2305.13528/) (2023)
- [Knowledge-Aware Audio-Grounded Generative Slot Filling for Limited Annotated Data](https://mlsanity.com/paper/2307.01764/) (2023)
- [Speech-based Slot Filling using Large Language Models](https://mlsanity.com/paper/2311.07418/) (2023)
- [SQATIN: Supervised Instruction Tuning Meets Question Answering for Improved Dialogue NLU](https://mlsanity.com/paper/2311.09502/) (2023)
- [MISE: Meta-knowledge Inheritance for Social Media-Based Stressor Estimation](https://mlsanity.com/paper/2505.03827/) (2025)
- [Towards Multi-Modal DBMSs for Seamless Querying of Texts and Tables](https://mlsanity.com/paper/2304.13559/) (2023)
- [Unifying Token and Span Level Supervisions for Few-Shot Sequence Labeling](https://mlsanity.com/paper/2307.07946/) (2023)
- [Leveraging Explicit Procedural Instructions for Data-Efficient Action Prediction](https://mlsanity.com/paper/2306.03959/) (2023)

## Abstract (arXiv)

We propose ConVEx (Conversational Value Extractor), an efficient pretraining and fine-tuning neural approach for slot-labeling dialog tasks. Instead of relying on more general pretraining objectives from prior work (e.g., language modeling, response selection), ConVEx's pretraining objective, a novel pairwise cloze task using Reddit data, is well aligned with its intended usage on sequence labeling tasks. This enables learning domain-specific slot labelers by simply fine-tuning decoding layers of the pretrained general-purpose sequence labeling model, while the majority of the pretrained model's parameters are kept frozen. We report state-of-the-art performance of ConVEx across a range of diverse domains and data sets for dialog slot-labeling, with the largest gains in the most challenging, few-shot setups. We believe that ConVEx's reduced pretraining times (i.e., only 18 hours on 12 GPUs) and cost, along with its efficient fine-tuning and strong performance, promise wider portability and scalability for data-efficient sequence-labeling tasks in general.