# Matching-based Term Semantics Pre-training for Spoken Patient Query Understanding

arXiv: https://arxiv.org/abs/2303.01341 · HTML: https://mlsanity.com/paper/2303.01341/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 2 Mar 2023.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Mining](https://mlsanity.com/paper/2210.10341/) (2022)
- [BERT for Joint Intent Classification and Slot Filling](https://mlsanity.com/paper/1902.10909/) (2019)
- [GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injection](https://mlsanity.com/paper/2111.14592/) (2021)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [BioBART: Pretraining and Evaluation of A Biomedical Generative Language Model](https://mlsanity.com/paper/2204.03905/) (2022)
- [1-bit Adam: Communication Efficient Large-Scale Training with Adam's Convergence Speed](https://mlsanity.com/paper/2102.02888/) (2021)
- [A Co-Interactive Transformer for Joint Slot Filling and Intent Detection](https://mlsanity.com/paper/2010.03880/) (2020)
- [Conversations Are Not Flat: Modeling the Dynamic Information Flow across Dialogue Utterances](https://mlsanity.com/paper/2106.02227/) (2021)
- [MixSpeech: Data Augmentation for Low-resource Automatic Speech Recognition](https://mlsanity.com/paper/2102.12664/) (2021)
- [Improving End-to-End Contextual Speech Recognition with Fine-Grained Contextual Knowledge Selection](https://mlsanity.com/paper/2201.12806/) (2022)
- [Pretraining Methods for Dialog Context Representation Learning](https://mlsanity.com/paper/1906.00414/) (2019)
- [Semi-Supervised Variational Reasoning for Medical Dialogue Generation](https://mlsanity.com/paper/2105.06071/) (2021)
- [SMedBERT: A Knowledge-Enhanced Pre-trained Language Model with Structured Semantics for Medical Text Mining](https://mlsanity.com/paper/2108.08983/) (2021)
- [CIF-based Collaborative Decoding for End-to-end Contextual Speech Recognition](https://mlsanity.com/paper/2012.09466/) (2020)
- [Structural Pre-training for Dialogue Comprehension](https://mlsanity.com/paper/2105.10956/) (2021)
- [ReMeDi: Resources for Multi-domain, Multi-service, Medical Dialogues](https://mlsanity.com/paper/2109.00430/) (2021)
- [Conceptualized Representation Learning for Chinese Biomedical Text Mining](https://mlsanity.com/paper/2008.10813/) (2020)
- [Dialogue-oriented Pre-training](https://mlsanity.com/paper/2106.00420/) (2021)
- [Investigation of Language Understanding Impact for Reinforcement Learning Based Dialogue Systems](https://mlsanity.com/paper/1703.07055/) (2017)
- [Towards Efficient Dialogue Pre-training with Transferable and Interpretable Latent Structure](https://mlsanity.com/paper/2210.12461/) (2022)
- [Exploiting Sentence and Context Representations in Deep Neural Models for Spoken Language Understanding](https://mlsanity.com/paper/1610.04120/) (2016)
- [Prompt-based Generative Approach towards Multi-Hierarchical Medical Dialogue State Tracking](https://mlsanity.com/paper/2203.09946/) (2022)

## Built on by (most cited first, 2 of 2)

- [A Knowledge-enhanced Two-stage Generative Framework for Medical Dialogue Information Extraction](https://mlsanity.com/paper/2307.16200/) (2023)
- [Locality enhanced dynamic biasing and sampling strategies for contextual ASR](https://mlsanity.com/paper/2401.13146/) (2024)

## Abstract (arXiv)

Medical Slot Filling (MSF) task aims to convert medical queries into structured information, playing an essential role in diagnosis dialogue systems. However, the lack of sufficient term semantics learning makes existing approaches hard to capture semantically identical but colloquial expressions of terms in medical conversations. In this work, we formalize MSF into a matching problem and propose a Term Semantics Pre-trained Matching Network (TSPMN) that takes both terms and queries as input to model their semantic interaction. To learn term semantics better, we further design two self-supervised objectives, including Contrastive Term Discrimination (CTD) and Matching-based Mask Term Modeling (MMTM). CTD determines whether it is the masked term in the dialogue for each given term, while MMTM directly predicts the masked ones. Experimental results on two Chinese benchmarks show that TSPMN outperforms strong baselines, especially in few-shot settings.