mlsanityData to 7 Oct 2026

Code-switched inspired losses for generic spoken dialog representations

arXiv 2108.12465 ↗First posted 27 Aug 2021cs.CL cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

Spoken dialog systems need to be able to handle both multiple languages and multilinguality inside a conversation (textite.g in case of code-switching). In this work, we introduce new pretraining losses tailored to learn multilingual spoken dialog representations. The goal of these losses is to expose the model to code-switched language. To scale up training, we automatically build a pretraining corpus composed of multilingual conversations in five different languages (French, Italian, English, German and Spanish) from textttOpenSubtitles, a huge multilingual corpus composed of 24.3G tokens. We test the generic representations on textttMIAM, a new benchmark composed of five dialog act corpora on the same aforementioned languages as well as on two novel multilingual downstream tasks (textiti.e multilingual mask utterance retrieval and multilingual inconsistency identification). Our experiments show that our new code switched-inspired losses achieve a better performance in both monolingual and multilingual settings.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 60 of its 115 references that are in the map. First posted on arXiv 27 Aug 2021.

Builds on

60 of 115 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  6. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  7. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2018SentencePiece: A simple and language independent subword tokenizer and detokenizer for Neural Text Processingcited by 1,973
38 more
  1. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  2. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  3. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  4. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  5. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  6. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,028
  7. 2019DialoGPT: Large-Scale Generative Pre-training for Conversational Response Generationcited by 842
  8. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  9. 2020The State and Fate of Linguistic Diversity and Inclusion in the NLP Worldcited by 789
  10. 2017DailyDialog: A Manually Labelled Multi-turn Dialogue Datasetcited by 649
  11. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 636
  12. 2020Language-agnostic BERT Sentence Embeddingcited by 621
  13. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  14. 2017A Structured Self-attentive Sentence Embeddingcited by 318
  15. 2020How Good is Your Tokenizer? On the Monolingual Performance of Multilingual Language Modelscited by 313
  16. 2020Evaluation of Text Generation: A Surveycited by 275
  17. 2013Exploiting Similarities among Languages for Machine Translationcited by 187
  18. 2019DialogueGCN: A Graph Convolutional Neural Network for Emotion Recognition in Conversationcited by 184
  19. 2019Cross-Lingual Ability of Multilingual BERT: An Empirical Studycited by 182
  20. 2015The Ubuntu Dialogue Corpus: A Large Dataset for Research in Unstructured Multi-Turn Dialogue Systemscited by 172
  21. 2019FlauBERT: Unsupervised Language Model Pre-training for Frenchcited by 153
  22. 2017A Survey Of Cross-lingual Word Embedding Modelscited by 142
  23. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  24. 2020GLUECoS : An Evaluation Benchmark for Code-Switched NLPcited by 101
  25. 2019ConveRT: Efficient and Accurate Conversational Representations from Transformerscited by 95
  26. 2021Are Multilingual Models Effective in Code-Switching?cited by 56
  27. 2019Evaluating Coherence in Dialogue Systems using Entailmentcited by 53
  28. 2019Affect-Driven Dialog Generationcited by 50
  29. 2019Do Neural Dialog Systems Use the Conversation History Effectively? An Empirical Studycited by 47
  30. 2021A Novel Estimator of Mutual Information for Learning to Disentangle Textual Representationscited by 37
  31. 2020Hierarchical Pre-training for Sequence Labelling in Spoken Dialogcited by 36
  32. 2019Pretraining Methods for Dialog Context Representation Learningcited by 35
  33. 2020Towards Unified Dialogue System Evaluation: A Comprehensive Analysis of Current Evaluation Protocolscited by 35
  34. 2021ProphetNet-X: Large-Scale Pre-training Models for English, Chinese, Multi-lingual, Dialog, and Code Generationcited by 31
  35. 2020Guiding attention in Sequence-to-sequence models for Dialogue Act predictioncited by 26
  36. 2021Code-Mixing on Sesame Street: Dawn of the Adversarial Polyglotscited by 23
  37. 2020Heavy-tailed Representations, Text Polarity Classification & Data Augmentationcited by 20
  38. 2018Zero-Shot Cross-lingual Classification Using Multilingual Neural Machine Translationcited by 19

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 5; 2023: 5.

10 papers.

  1. 2022Of Human Criteria and Automatic Metrics: A Benchmark of the Evaluation of Story Generationcited by 73
  2. 2022Rainproof: An Umbrella To Shield Text Generators From Out-Of-Distribution Datacited by 21
  3. 2023Unsupervised Layer-wise Score Aggregation for Textual OOD Detectioncited by 15
  4. 2022Learning Disentangled Textual Representations via Statistical Measures of Similaritycited by 14
  5. 2023Transductive Learning for Textual Few-Shot Classification in API-based Embedding Modelscited by 4
  6. 2022Beyond Mahalanobis-Based Scores for Textual OOD Detectioncited by 3
  7. 2023Toward Stronger Textual Attack Detectorscited by 3
  8. 2023Sequence-to-Sequence Spanish Pre-trained Language Modelscited by 2
  9. 2023A Novel Information-Theoretic Objective to Disentangle Representations for Fair Classificationcited by 1
  10. 2022What are the best systems? New perspectives on NLP Benchmarkingcited by 0