mlsanityData to 7 Oct 2026

Duplex Conversation: Towards Human-like Interaction in Spoken Dialogue Systems

arXiv 2205.15060 ↗First posted 30 May 2022cs.CLcited by 28

Not on the map. Why?

Abstract (arXiv)

In this paper, we present Duplex Conversation, a multi-turn, multimodal spoken dialogue system that enables telephone-based agents to interact with customers like a human. We use the concept of full-duplex in telecommunication to demonstrate what a human-like interactive experience should be and how to achieve smooth turn-taking through three subtasks: user state detection, backchannel selection, and barge-in detection. Besides, we propose semi-supervised learning with multimodal data augmentation to leverage unlabeled data to increase model generalization. Experimental results on three sub-tasks show that the proposed method achieves consistent improvements compared with baselines. We deploy the Duplex Conversation to Alibaba intelligent customer service and share lessons learned in production. Online A/B experiments show that the proposed system can significantly reduce response latency by 50%.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 35 of its 72 references that are in the map. First posted on arXiv 30 May 2022.

Builds on

35 of 72 references.

References · 35

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  2. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  3. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  4. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  5. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  6. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  7. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  8. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  9. 2015MUSAN: A Music, Speech, and Noise Corpuscited by 400
  10. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  11. 2020MISA: Modality-Invariant and -Specific Representations for Multimodal Sentiment Analysiscited by 262
  12. 2021Learning Modality-Specific Representations with Self-Supervised Multi-Task Learning for Multimodal Sentiment Analysiscited by 179
23 more
  1. 2018The Design and Implementation of XiaoIce, an Empathetic Social Chatbotcited by 172
  2. 2021GALAXY: A Generative Pre-trained Model for Task-Oriented Dialog with Semi-Supervised Learning and Explicit Policy Injectioncited by 102
  3. 2019A Pre-training Based Personalized Dialogue Generation Model with Persona-sparse Datacited by 74
  4. 2020Discovering New Intents with Deep Aligned Clusteringcited by 68
  5. 2019Deep Unknown Intent Detection with Margin Losscited by 66
  6. 2019Induction Networks for Few-Shot Text Classificationcited by 64
  7. 2019Discovering New Intents via Constrained Deep Adaptive Clustering with Cluster Refinementcited by 56
  8. 2020Deep Open Intent Classification with Adaptive Decision Boundarycited by 56
  9. 2020TurnGPT: a Transformer-based Language Model for Predicting Turn-taking in Spoken Dialogcited by 55
  10. 2022S2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parserscited by 41
  11. 2021Preview, Attend and Review: Schema-Aware Curriculum Learning for Multi-Domain Dialog State Trackingcited by 35
  12. 2021EVA: An Open-Domain Chinese Dialogue System with Large-Scale Generative Pre-Trainingcited by 34
  13. 2021MMChat: Multi-Modal Chat Dataset on Social Mediacited by 30
  14. 2021Diversifying Dialog Generation via Adaptive Label Smoothingcited by 25
  15. 2021TEXTOIR: An Integrated and Visualized Platform for Text Open Intent Recognitioncited by 24
  16. 2020Dynamic Memory Induction Networks for Few-Shot Text Classificationcited by 23
  17. 2020Dialogue Distillation: Open-Domain Dialogue Augmentation Using Unpaired Datacited by 20
  18. 2021DialogueCSE: Dialogue-based Contrastive Learning of Sentence Embeddingscited by 19
  19. 2020Spoken Language Interaction with Robots: Research Issues and Recommendations, Report from the NSF Future Directions Workshopcited by 15
  20. 2021Revisiting the Boundary between ASR and NLU in the Age of Conversational Dialog Systemscited by 15
  21. 2022A Slot Is Not Built in One Utterance: Spoken Language Dialogs with Sub-Slotscited by 13
  22. 2020A Post-processing Method for Detecting Unknown Intent of Dialogue System via Pre-trained Deep Neural Network Classifiercited by 11
  23. 2022Improving Meta-learning for Low-resource Text Classification and Generation via Memory Imitationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 7; 2024: 3; 2025: 10; 2026: 5.

28 papers.

Sort:

By area:

  1. 2024Language Model Can Listen While Speakingcited by 62
  2. 2022UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognitioncited by 61
  3. 2024A Full-duplex Speech Dialogue Scheme Based On Large Language Modelscited by 40
  4. 2023Transformers in Speech Processing: A Surveycited by 38
  5. 2022Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsingcited by 30
  6. 2023UniSA: Unified Generative Framework for Sentiment Analysiscited by 17
  7. 2024SSR: Alignment-Aware Modality Connector for Speech Language Modelscited by 16
  8. 2025FlexDuo: A Pluggable System for Enabling Full-Duplex Capabilities in Speech Dialogue Systemscited by 15
  9. 2023Speech-Text Dialog Pre-training for Spoken Dialog Understanding with Explicit Cross-Modal Alignmentcited by 13
  10. 2025Applying General Turn-taking Models to Conversational Human-Robot Interactioncited by 13
  11. 2025Chronological Thinking in Full-Duplex Spoken Dialogue Language Modelscited by 13
  12. 2023Unsupervised Dialogue Topic Segmentation with Topic-aware Utterance Representationcited by 11
16 more
  1. 2025Easy Turn: Integrating Acoustic and Linguistic Modalities for Robust Turn-Taking in Full-Duplex Spoken Dialogue Systemscited by 11
  2. 2025OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-Time Self-Aware Emotional Speech Synthesiscited by 10
  3. 2022FCTalker: Fine and Coarse Grained Context Modeling for Expressive Conversational Speech Synthesiscited by 9
  4. 2025LLM-Enhanced Dialogue Management for Full-Duplex Spoken Dialogue Systemscited by 9
  5. 2023Empathetic Response Generation via Emotion Cause Transition Graphcited by 8
  6. 2025Voila: Voice-Language Foundation Models for Real-Time Autonomous Interaction and Voice Role-Playcited by 8
  7. 2023Self-Explanation Prompting Improves Dialogue Understanding in Large Language Modelscited by 5
  8. 2025Telephone Surveys Meet Conversational AI: Evaluating a LLM-Based Telephone Survey System at Scalecited by 5
  9. 2025Speculative End-Turn Detector for Efficient Speech Chatbot Assistantcited by 5
  10. 2025Affective and Conversational Predictors of Re-Engagement in Human-Robot Interactions -- A Student-Centered Study with A Humanoid Social Robotcited by 3
  11. 2026Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMscited by 3
  12. 2023Towards Objective Evaluation of Socially-Situated Conversational Robots: Assessing Human-Likeness through Multimodal User Behaviorscited by 0
  13. 2026From Transcripts to AI Agents: Knowledge Extraction, RAG Integration, and Robust Evaluation of Conversational AI Assistantscited by 0
  14. 2026Does Listening Matter? Backchanneling and Nodding in AI Clonecited by 0
  15. 2026Training Intelligent Voice Assistant Wakeup with Controllable Synthetic Conversationscited by 0
  16. 2026Neither Silence nor Overlap Is Failure: Intent-Conditioned Evaluation of Turn-Taking in Full-Duplex Spoken Dialogue Modelscited by 0