# Task Conditioned BERT for Joint Intent Detection and Slot-filling

arXiv: https://arxiv.org/abs/2308.06165 · HTML: https://mlsanity.com/paper/2308.06165/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 26 references that are in the map.
- First posted on arXiv 11 Aug 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Towards Scalable Multi-domain Conversational Agents: The Schema-Guided Dialogue Dataset](https://mlsanity.com/paper/1909.05855/) (2019)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [A Simple Language Model for Task-Oriented Dialogue](https://mlsanity.com/paper/2005.00796/) (2020)
- [Transferable Multi-Domain State Generator for Task-Oriented Dialogue Systems](https://mlsanity.com/paper/1905.08743/) (2019)
- [MultiWOZ 2.2 : A Dialogue Dataset with Additional Annotation Corrections and State Tracking Baselines](https://mlsanity.com/paper/2007.12720/) (2020)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [TOD-BERT: Pre-trained Natural Language Understanding for Task-Oriented Dialogue](https://mlsanity.com/paper/2004.06871/) (2020)
- [TripPy: A Triple Copy Strategy for Value Independent Neural Dialog State Tracking](https://mlsanity.com/paper/2005.02877/) (2020)
- [A Stack-Propagation Framework with Token-Level Intent Detection for Spoken Language Understanding](https://mlsanity.com/paper/1909.02188/) (2019)
- [Find or Classify? Dual Strategy for Slot-Value Predictions on Multi-Domain Dialog State Tracking](https://mlsanity.com/paper/1910.03544/) (2019)
- [MultiWOZ 2.3: A multi-domain task-oriented dialogue dataset enhanced with annotation corrections and co-reference annotation](https://mlsanity.com/paper/2010.05594/) (2020)
- [SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2010.02693/) (2020)
- [BERT-DST: Scalable End-to-End Dialogue State Tracking with Bidirectional Encoder Representations from Transformer](https://mlsanity.com/paper/1907.03040/) (2019)
- [Annotation Inconsistency and Entity Bias in MultiWOZ](https://mlsanity.com/paper/2105.14150/) (2021)
- [An End-to-end Approach for Handling Unknown Slot Values in Dialogue State Tracking](https://mlsanity.com/paper/1805.01555/) (2018)
- [Jointly Optimizing State Operation Prediction and Value Generation for Dialogue State Tracking](https://mlsanity.com/paper/2010.14061/) (2020)
- [Improving Slot Filling by Utilizing Contextual Information](https://mlsanity.com/paper/1911.01680/) (2019)
- [ShopTalk: A System for Conversational Faceted Search](https://mlsanity.com/paper/2109.00702/) (2021)

## Built on by (most cited first, 3 of 3)

- [The Wizard of Curiosities: Enriching Dialogues with Fun Facts](https://mlsanity.com/paper/2309.11283/) (2023)
- [TWIZ-v2: The Wizard of Multimodal Conversational-Stimulus](https://mlsanity.com/paper/2310.02118/) (2023)
- [From Large to Super-Tiny: End-to-End Optimization for Cost-Efficient LLMs](https://mlsanity.com/paper/2504.13471/) (2025)

## Abstract (arXiv)

Dialogue systems need to deal with the unpredictability of user intents to track dialogue state and the heterogeneity of slots to understand user preferences. In this paper we investigate the hypothesis that solving these challenges as one unified model will allow the transfer of parameter support data across the different tasks. The proposed principled model is based on a Transformer encoder, trained on multiple tasks, and leveraged by a rich input that conditions the model on the target inferences. Conditioning the Transformer encoder on multiple target inferences over the same corpus, i.e., intent and multiple slot types, allows learning richer language interactions than a single-task model would be able to. In fact, experimental results demonstrate that conditioning the model on an increasing number of dialogue inference tasks leads to improved results: on the MultiWOZ dataset, the joint intent and slot detection can be improved by 3.2% by conditioning on intent, 10.8% by conditioning on slot and 14.4% by conditioning on both intent and slots. Moreover, on real conversations with Farfetch costumers, the proposed conditioned BERT can achieve high joint-goal and intent detection performance throughout a dialogue.