mlsanityData to 14 Oct 2026

Towards Human-like Multimodal Conversational Agent by Generating Engaging Speech

arXiv 2509.14627 ↗First posted 18 Sep 2025cs.HC cs.AI cs.CLcited by 0

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

Human conversation involves language, speech, and visual cues, with each medium providing complementary information. For instance, speech conveys a vibe or tone not fully captured by text alone. While multimodal LLMs focus on generating text responses from diverse inputs, less attention has been paid to generating natural and engaging speech. We propose a human-like agent that generates speech responses based on conversation mood and responsive style information. To achieve this, we build a novel MultiSensory Conversation dataset focused on speech to enable agents to generate natural speech. We then propose a multimodal LLM-based model for generating text responses and voice descriptions, which are used to generate speech covering paralinguistic information. Experimental results demonstrate the effectiveness of utilizing both visual and audio modalities in conversation to generate engaging speech. The source code is available in https://github.com/kimtaesu24/MSenC

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 25 of its 31 references that are in the map. First posted on arXiv 18 Sep 2025.

Builds on

25 of 31 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  7. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  8. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  9. 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
  10. 2023Mistral 7Bcited by 3,164
  11. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  12. 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
13 more
  1. 2023Video-LLaMA: An Instruction-tuned Audio-Visual Language Model for Video Understandingcited by 1,471
  2. 2018MELD: A Multimodal Multi-Party Dataset for Emotion Recognition in Conversationscited by 643
  3. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 615
  4. 2023SpeechGPT: Empowering Large Language Models with Intrinsic Cross-Modal Conversational Abilitiescited by 529
  5. 2023NaturalSpeech 2: Latent Diffusion Models are Natural and Zero-Shot Speech and Singing Synthesizerscited by 235
  6. 2024AnyGPT: Unified Multimodal LLM with Discrete Sequence Modelingcited by 217
  7. 2023StyleTTS 2: Towards Human-Level Text-to-Speech through Style Diffusion and Adversarial Training with Large Speech Language Modelscited by 152
  8. 2022Wespeaker: A Research and Production oriented Speaker Embedding Learning Toolkitcited by 97
  9. 2024Natural language guidance of high-fidelity text-to-speech with synthetic annotationscited by 67
  10. 2023CoDi-2: In-Context, Interleaved, and Interactive Any-to-Any Generationcited by 63
  11. 2024Let's Go Real Talk: Spoken Dialogue Model for Face-to-Face Conversationcited by 41
  12. 2023HierSpeech++: Bridging the Gap between Semantic and Acoustic Representation of Speech by Hierarchical Variational Inference for Zero-shot Speech Synthesiscited by 35
  13. 2018A Face-to-Face Neural Conversation Modelcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Talk With Human-like Agents: Empathetic Dialogue Through Perceptible Acoustic Reception and Reaction
  2. 2023Towards human-like spoken dialogue generation between AI agents from written dialogue
  3. 2025Beyond Words: Multimodal LLM Knows When to Speak
  4. 2024Multimodal Human-Autonomous Agents Interaction Using Pre-Trained Language and Visual Foundation Models
  5. 2025Spoken Conversational Agents with Large Language Models
  6. 2024SpeechAgents: Human-Communication Simulation with Multi-Modal Multi-Agent Systems
  7. 2026Conversational Voice Aesthetic Model with Reinforcement Learning from Human Listeners
  8. 2025Speaking Beyond Language: A Large-Scale Multimodal Dataset for Learning Nonverbal Cues from Video-Grounded Dialogues