mlsanityData to 7 Oct 2026

Hierarchical Pre-training for Sequence Labelling in Spoken Dialog

arXiv 2009.11152 ↗First posted 23 Sep 2020cs.CL cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Sequence labelling tasks like Dialog Act and Emotion/Sentiment identification are a key component of spoken dialog systems. In this work, we propose a new approach to learn generic representations adapted to spoken dialog, which we evaluate on a new benchmark we call Sequence labellIng evaLuatIon benChmark fOr spoken laNguagE benchmark (textttSILICONE). textttSILICONE is model-agnostic and contains 10 different datasets of various sizes. We obtain our representations with a hierarchical encoder based on transformer architectures, for which we extend two well-known pre-training objectives. Pre-training is performed on OpenSubtitles: a large corpus of spoken dialog containing over 2.3 billion of tokens. We demonstrate how hierarchical encoders achieve competitive results with consistently fewer parameters compared to state-of-the-art models and we show their importance for both pre-training and fine-tuning.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 23 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 8; 2022: 12; 2023: 11; 2024: 3; 2025: 2.

36 papers.

Sort:

By area:

  1. 2022XAI for Transformers: Better Explanations through Conservative Propagationcited by 100
  2. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  3. 2021Automatic Text Evaluation through the Lens of Wasserstein Barycenterscited by 38
  4. 2023Transformers in Speech Processing: A Surveycited by 38
  5. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  6. 2021InfoLM: A New Metric to Evaluate Summarization & Data2Text Generationcited by 33
  7. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  8. 2024Stress-Testing Long-Context Language Models with Lifelong ICL and Task Haystackcited by 19
  9. 2021A Pseudo-Metric between Probability Distributions based on Depth-Trimmed Regionscited by 17
  10. 2021Improving Multimodal fusion via Mutual Dependency Maximisationcited by 15
  11. 2023Unsupervised Layer-wise Score Aggregation for Textual OOD Detectioncited by 15
  12. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
24 more
  1. 2023Cluster-Level Contrastive Learning for Emotion Recognition in Conversationscited by 14
  2. 2023Towards More Robust NLP System Evaluation: Handling Missing Scores in Benchmarkscited by 14
  3. 2021Beam Search with Bidirectional Strategies for Neural Response Generationcited by 12
  4. 2022Deep Emotion Recognition in Textual Conversations: A Surveycited by 12
  5. 2021Code-switched inspired losses for generic spoken dialog representationscited by 10
  6. 2021Few-Shot Emotion Recognition in Conversation with Sequential Prototypical Networkscited by 10
  7. 2022The Glass Ceiling of Automatic Evaluation in Natural Language Generationcited by 10
  8. 2023Massively Multilingual Corpus of Sentiment Datasets and Multi-faceted Sentiment Classification Benchmarkcited by 10
  9. 2023Hierarchical Dialogue Understanding with Special Tokens and Turn-level Attentioncited by 6
  10. 2023Disentangled Variational Autoencoder for Emotion Recognition in Conversationscited by 5
  11. 2023Transductive Learning for Textual Few-Shot Classification in API-based Embedding Modelscited by 4
  12. 2022Learning-by-Narrating: Narrative Pre-Training for Zero-Shot Dialogue Comprehensioncited by 3
  13. 2022Assessment of Massively Multilingual Sentiment Classifierscited by 3
  14. 2022Multi-Task Learning Framework for Extracting Emotion Cause Span and Entailment in Conversationscited by 3
  15. 2022Beyond Mahalanobis-Based Scores for Textual OOD Detectioncited by 3
  16. 2023Toward Stronger Textual Attack Detectorscited by 3
  17. 2024A Strategy to Combine 1stGen Transformers and Open LLMs for Automatic Text Classificationcited by 3
  18. 2023Dialogue Agents 101: A Beginner's Guide to Critical Ingredients for Designing Effective Conversational Systemscited by 1
  19. 2023A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classificationcited by 1
  20. 2024Evaluating the fairness of task-adaptive pretraining on unlabeled test data before few-shot text classificationcited by 1
  21. 2022What are the best systems? New perspectives on NLP Benchmarkingcited by 0
  22. 2022A survey of neural models for the automatic analysis of conversation: Towards a better integration of the social sciencescited by 0
  23. 2025Learning LLM Preference over Intra-Dialogue Pairs: A Framework for Utterance-level Understandingscited by 0
  24. 2025textttDIAMONDs: A Dataset for mathbbDynamic mathbbInformation mathbbAnd mathbbMental modeling mathbbOf mathbbNumeric mathbbDiscussionscited by 0