# S2Dialog: Multimodal Dialogue Retrieval with Semantic and Acoustic-Style Modeling

arXiv: https://arxiv.org/abs/2608.14029 · HTML: https://mlsanity.com/paper/2608.14029/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 41 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Builds on (most cited first, 30 of 32)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection](https://mlsanity.com/paper/2310.11511/) (2023)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [In-Context Retrieval-Augmented Language Models](https://mlsanity.com/paper/2302.00083/) (2023)
- [Keystroke Dynamics as Part of Lifelogging](https://mlsanity.com/paper/2010.10144/) (2020)
- [One Embedder, Any Task: Instruction-Finetuned Text Embeddings](https://mlsanity.com/paper/2212.09741/) (2022)
- [Kimi-Audio Technical Report](https://mlsanity.com/paper/2504.18425/) (2025)
- [The Design and Implementation of XiaoIce, an Empathetic Social Chatbot](https://mlsanity.com/paper/1812.08989/) (2018)
- [StyleDiffusion: Controllable Disentangled Style Transfer via Diffusion Models](https://mlsanity.com/paper/2308.07863/) (2023)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [DailyTalk: Spoken Dialogue Dataset for Conversational Text-to-Speech](https://mlsanity.com/paper/2207.01063/) (2022)
- [Deep Learning using Rectified Linear Units (ReLU)](https://mlsanity.com/paper/1803.08375/) (2018)
- [Boosting Conversational Question Answering with Fine-Grained Retrieval-Augmentation and Self-Check](https://mlsanity.com/paper/2403.18243/) (2024)
- [Retrieval Augmented End-to-End Spoken Dialog Models](https://mlsanity.com/paper/2402.01828/) (2024)
- [CLAPSpeech: Learning Prosody from Text Context with Contrastive Language-Audio Pre-training](https://mlsanity.com/paper/2305.10763/) (2023)
- [Effective and Efficient Conversation Retrieval for Dialogue State Tracking with Implicit Text Summaries](https://mlsanity.com/paper/2402.13043/) (2024)
- [Chain-Talker: Chain Understanding and Rendering for Empathetic Conversational Speech Synthesis](https://mlsanity.com/paper/2505.12597/) (2025)
- [Emphasis Rendering for Conversational Text-to-Speech with Multi-modal Multi-scale Context Modeling](https://mlsanity.com/paper/2410.09524/) (2024)
- [DFA-RAG: Conversational Semantic Router for Large Language Model with Definite Finite Automaton](https://mlsanity.com/paper/2402.04411/) (2024)
- [Intra- and Inter-modal Context Interaction Modeling for Conversational Speech Synthesis](https://mlsanity.com/paper/2412.18733/) (2024)
- [History-Aware Conversational Dense Retrieval](https://mlsanity.com/paper/2401.16659/) (2024)
- [Retrieval-Augmented Dialogue Knowledge Aggregation for Expressive Conversational Speech Synthesis](https://mlsanity.com/paper/2501.06467/) (2025)
- [Hearing More with Less: Multi-Modal Retrieval-and-Selection Augmented Conversational LLM-Based ASR](https://mlsanity.com/paper/2508.01166/) (2025)
- [A retrieval-based dialogue system utilizing utterance and context embeddings](https://mlsanity.com/paper/1710.05780/) (2017)
- [UniRetriever: Multi-task Candidates Selection for Various Context-Adaptive Conversational Retrieval](https://mlsanity.com/paper/2402.16261/) (2024)
- [AMNS: Attention-Weighted Selective Mask and Noise Label Suppression for Text-to-Image Person Retrieval](https://mlsanity.com/paper/2409.06385/) (2024)

## Abstract (arXiv)

Multimodal dialogue retrieval aims to retrieve dialogues from multimodal dialogue banks that are similar to a target dialogue in terms of both textual semantics and acoustic conversational styles. Such dialogue-level retrieval is crucial for many dialogue-related tasks, including Emotion Recognition in Conversation, Spoken Dialogue Systems, and Conversational Speech Synthesis, where external dialogue examples can provide valuable semantic and stylistic references. However, existing retrieval methods are still largely limited to utterance-level or unimodal matching, and often fail to capture the global semantic coherence and stylistic consistency of an entire dialogue. To address this gap, we propose S2Dialog, a unified framework for dialogue-level semantic-style retrieval from multimodal dialogue banks. Specifically, S2Dialog consists of a Dialogue-level Textual Retriever and a Dialogue-level Acoustic Retriever, which encode the textual and acoustic modalities of a dialogue into dialogue-level representations, respectively. To further enhance multimodal retrieval, we introduce Dialogue-level Textual-Acoustic Contrastive Learning, which aligns semantically and stylistically similar dialogues while distinguishing unrelated ones. Extensive experiments on the multimodal dialogue dataset DailyTalk demonstrate that S2Dialog achieves outstanding retrieval performance.