# PSSAT: A Perturbed Semantic Structure Awareness Transferring Method for Perturbation-Robust Slot Filling

arXiv: https://arxiv.org/abs/2208.11508 · HTML: https://mlsanity.com/paper/2208.11508/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 84 references that are in the map.
- First posted on arXiv 24 Aug 2022.

## Builds on (most cited first, 30 of 50)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [MultiWOZ -- A Large-Scale Multi-Domain Wizard-of-Oz Dataset for Task-Oriented Dialogue Modelling](https://mlsanity.com/paper/1810.00278/) (2018)
- [Neural Architectures for Named Entity Recognition](https://mlsanity.com/paper/1603.01360/) (2016)
- [Snips Voice Platform: an embedded Spoken Language Understanding system for private-by-design voice interfaces](https://mlsanity.com/paper/1805.10190/) (2018)
- [Synthetic and Natural Noise Both Break Neural Machine Translation](https://mlsanity.com/paper/1711.02173/) (2017)
- [ConSERT: A Contrastive Framework for Self-Supervised Sentence Representation Transfer](https://mlsanity.com/paper/2105.11741/) (2021)
- [Named Entity Recognition with Bidirectional LSTM-CNNs](https://mlsanity.com/paper/1511.08308/) (2015)
- [Attention-Based Recurrent Neural Network Models for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1609.01454/) (2016)
- [Yara Parser: A Fast and Accurate Dependency Parser](https://mlsanity.com/paper/1503.06733/) (2015)
- [Evaluating the Robustness of Neural Language Models to Input Perturbations](https://mlsanity.com/paper/2108.12237/) (2021)
- [A Novel Bi-directional Interrelated Model for Joint Intent Detection and Slot Filling](https://mlsanity.com/paper/1907.00390/) (2019)
- [A Simple but Tough-to-Beat Data Augmentation Approach for Natural Language Understanding and Generation](https://mlsanity.com/paper/2009.13818/) (2020)
- [Sequence-to-Sequence Data Augmentation for Dialogue Language Understanding](https://mlsanity.com/paper/1807.01554/) (2018)
- [Text Data Augmentation Made Simple By Leveraging NLP Cloud APIs](https://mlsanity.com/paper/1812.04718/) (2018)
- [RADDLE: An Evaluation Benchmark and Analysis Platform for Robust Task-oriented Dialog Systems](https://mlsanity.com/paper/2012.14666/) (2020)
- [Adversarial Diversity and Hard Positive Generation](https://mlsanity.com/paper/1605.01775/) (2016)
- [Rethinking Generalization of Neural Models: A Named Entity Recognition Case Study](https://mlsanity.com/paper/2001.03844/) (2020)
- [Robustness Testing of Language Understanding in Task-Oriented Dialog](https://mlsanity.com/paper/2012.15262/) (2020)
- [InstructionNER: A Multi-Task Instruction-Based Generative Framework for Few-shot NER](https://mlsanity.com/paper/2203.03903/) (2022)
- [Joint Contextual Modeling for ASR Correction and Language Understanding](https://mlsanity.com/paper/2002.00750/) (2020)
- [Defending Pre-trained Language Models from Adversarial Word Substitutions Without Performance Sacrifice](https://mlsanity.com/paper/2105.14553/) (2021)
- [RockNER: A Simple Method to Create Adversarial Examples for Evaluating the Robustness of Named Entity Recognition Models](https://mlsanity.com/paper/2109.05620/) (2021)
- [Conditional Augmentation for Aspect Term Extraction via Masked Sequence-to-Sequence Generation](https://mlsanity.com/paper/2004.14769/) (2020)
- [How Robust Are Character-Based Word Embeddings in Tagging and MT Against Wrod Scramlbing or Randdm Nouse?](https://mlsanity.com/paper/1704.04441/) (2017)
- [Phoneme-BERT: Joint Language Modelling of Phoneme Sequence and ASR Transcript](https://mlsanity.com/paper/2102.00804/) (2021)

## Built on by (most cited first, 9 of 9)

- [Understand What LLM Needs: Dual Preference Alignment for Retrieval-Augmented Generation](https://mlsanity.com/paper/2406.18676/) (2024)
- [Revisit Input Perturbation Problems for LLMs: A Unified Robustness Evaluation Framework for Noisy Slot Filling Task](https://mlsanity.com/paper/2310.06504/) (2023)
- [Bridging the KB-Text Gap: Leveraging Structured Knowledge-aware Pre-training for KBQA](https://mlsanity.com/paper/2308.14436/) (2023)
- [Large Language Models Meet Open-World Intent Discovery and Recognition: An Evaluation of ChatGPT](https://mlsanity.com/paper/2310.10176/) (2023)
- [A Multi-Task Semantic Decomposition Framework with Task-specific Pre-training for Few-Shot NER](https://mlsanity.com/paper/2308.14533/) (2023)
- [Towards Robust and Generalizable Training: An Empirical Study of Noisy Slot Filling for Input Perturbations](https://mlsanity.com/paper/2310.03518/) (2023)
- [Watch the Speakers: A Hybrid Continuous Attribution Network for Emotion Recognition in Conversation With Emotion Disentanglement](https://mlsanity.com/paper/2309.09799/) (2023)
- [Noise-BERT: A Unified Perturbation-Robust Framework with Noise Alignment Pre-training for Noisy Slot Filling Task](https://mlsanity.com/paper/2402.14494/) (2024)
- [Towards Spoken Language Understanding via Multi-level Multi-grained Contrastive Learning](https://mlsanity.com/paper/2405.20852/) (2024)

## Abstract (arXiv)

Most existing slot filling models tend to memorize inherent patterns of entities and corresponding contexts from training data. However, these models can lead to system failure or undesirable outputs when being exposed to spoken language perturbation or variation in practice. We propose a perturbed semantic structure awareness transferring method for training perturbation-robust slot filling models. Specifically, we introduce two MLM-based training strategies to respectively learn contextual semantic structure and word distribution from unsupervised language perturbation corpus. Then, we transfer semantic knowledge learned from upstream training procedure into the original samples and filter generated data by consistency processing. These procedures aim to enhance the robustness of slot filling models. Experimental results show that our method consistently outperforms the previous basic methods and gains strong generalization while preventing the model from memorizing inherent patterns of entities and contexts.