mlsanityData to 7 Oct 2026

Slot Induction via Pre-trained Language Model Probing and Multi-level Contrastive Learning

arXiv 2308.04712 ↗First posted 9 Aug 2023cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Recent advanced methods in Natural Language Understanding for Task-oriented Dialogue (TOD) Systems (e.g., intent detection and slot filling) require a large amount of annotated data to achieve competitive performance. In reality, token-level annotations (slot labels) are time-consuming and difficult to acquire. In this work, we study the Slot Induction (SI) task whose objective is to induce slot boundaries without explicit knowledge of token-level slot annotations. We propose leveraging Unsupervised Pre-trained Language Model (PLM) Probing and Contrastive Learning mechanism to exploit (1) unsupervised semantic knowledge extracted from PLM, and (2) additional sentence-level intent label signals available from TOD. Our approach is shown to be effective in SI task and capable of bridging the gaps with token-level supervised models on two NLU benchmark datasets. When generalized to emerging intents, our SI objectives also provide enhanced slot label representations, leading to improved performance on the Slot Filling tasks.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 30 of its 50 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

30 of 50 references.

References · 30

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  5. 2020Approximate Nearest Neighbor Negative Contrastive Learning for Dense Text Retrievalcited by 846
  6. 2019SpanBERT: Improving Pre-training by Representing and Predicting Spanscited by 832
  7. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  8. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  9. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  10. 2018Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfacescited by 376
  11. 2020Perturbed Masking: Parameter-free Probing for Analyzing and Interpreting BERTcited by 109
  12. 2019A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Fillingcited by 85
18 more
  1. 2018Joint Slot Filling and Intent Detection via Capsule Neural Networkscited by 73
  2. 2017Automated Phrase Mining from Massive Text Corporacited by 70
  3. 2018Learning Named Entity Tagger using Domain-Specific Dictionarycited by 65
  4. 2018Zero-shot User Intent Detection via Capsule Neural Networkscited by 54
  5. 2020Recent Neural Methods on Slot Filling and Intent Classification for Task-Oriented Dialogue Systems: A Surveycited by 51
  6. 2020Are Pre-trained Language Models Aware of Phrases? Simple but Strong Baselines for Grammar Inductioncited by 47
  7. 2020SlotRefine: A Fast Non-Autoregressive Model for Joint Intent Detection and Slot Fillingcited by 42
  8. 2021Dense Hierarchical Retrieval for Open-Domain Question Answeringcited by 33
  9. 2021Robust Retrieval Augmented Generation for Zero-shot Slot Fillingcited by 24
  10. 2020CG-BERT: Conditional Text Generation with BERT for Generalized Few-shot Intent Detectioncited by 23
  11. 2021Generalized Zero-shot Intent Detection via Commonsense Knowledgecited by 21
  12. 2020Dynamic Semantic Matching and Aggregation Network for Few-shot Intent Detectioncited by 17
  13. 2021UCPhrase: Unsupervised Context-aware Quality Phrase Taggingcited by 14
  14. 2022GL-CLeF: A Global-Local Contrastive Learning Framework for Cross-lingual Spoken Language Understandingcited by 14
  15. 2021Automatic Intent-Slot Induction for Dialogue Systemscited by 13
  16. 2022Unsupervised Slot Schema Induction for Task-oriented Dialogcited by 11
  17. 2023Enhancing Cross-lingual Transfer via Phonemic Transcription Integrationcited by 11
  18. 2021CSS-LM: A Contrastive Framework for Semi-supervised Fine-tuning of Pre-trained Language Modelscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 3.

4 papers.

  1. 2023CoF-CoT: Enhancing Large Language Models with Coarse-to-Fine Chain-of-Thought Prompting for Multi-domain NLU Taskscited by 18
  2. 2024A Survey of Ontology Expansion for Conversational Understandingcited by 6
  3. 2024Prompting with Phonemes: Enhancing LLMs' Multilinguality for Non-Latin Script Languagescited by 6
  4. 2024CORI: CJKV Benchmark with Romanization Integration -- A step towards Cross-lingual Transfer Beyond Textual Scriptscited by 4