# Encoding Syntactic Knowledge in Transformer Encoder for Intent Detection and Slot Filling

arXiv: https://arxiv.org/abs/2012.11689 · HTML: https://mlsanity.com/paper/2012.11689/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Dec 2020.

## Built on by (most cited first, 10 of 10)

- [Intent Detection and Slot Filling for Vietnamese](https://mlsanity.com/paper/2104.02021/) (2021)
- [A Survey of Ontology Expansion for Conversational Understanding](https://mlsanity.com/paper/2410.15019/) (2024)
- [Explainable Slot Type Attentions to Improve Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2210.10227/) (2022)
- [Semi-Supervised Few-Shot Intent Classification and Slot Filling](https://mlsanity.com/paper/2109.08754/) (2021)
- [Intent Detection and Slot Filling for Home Assistants: Dataset and Analysis for Bangla and Sylheti](https://mlsanity.com/paper/2310.10935/) (2023)
- [Improving Dialectal Slot and Intent Detection with Auxiliary Tasks: A Multi-Dialectal Bavarian Case Study](https://mlsanity.com/paper/2501.03863/) (2025)
- [MIDAS: Multi-level Intent, Domain, And Slot Knowledge Distillation for Multi-turn NLU](https://mlsanity.com/paper/2408.08144/) (2024)
- [ESIE-BERT: Enriching Sub-words Information Explicitly with BERT for Joint Intent Classification and SlotFilling](https://mlsanity.com/paper/2211.14829/) (2022)
- [Requirements Elicitation in Cognitive Service for Recommendation](https://mlsanity.com/paper/2203.14958/) (2022)
- [Multi-grained Label Refinement Network with Dependency Structures for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/2209.04156/) (2022)

## Abstract (arXiv)

We propose a novel Transformer encoder-based architecture with syntactical knowledge encoded for intent detection and slot filling. Specifically, we encode syntactic knowledge into the Transformer encoder by jointly training it to predict syntactic parse ancestors and part-of-speech of each token via multi-task learning. Our model is based on self-attention and feed-forward layers and does not require external syntactic information to be available at inference time. Experiments show that on two benchmark datasets, our models with only two Transformer encoder layers achieve state-of-the-art results. Compared to the previously best performed model without pre-training, our models achieve absolute F1 score and accuracy improvement of 1.59% and 0.85% for slot filling and intent detection on the SNIPS dataset, respectively. Our models also achieve absolute F1 score and accuracy improvement of 0.1% and 0.34% for slot filling and intent detection on the ATIS dataset, respectively, over the previously best performed model. Furthermore, the visualization of the self-attention weights illustrates the benefits of incorporating syntactic information during training.