Factor-Conditioned Speaking-Style Captioning
Not on the map. Why?
Abstract (arXiv)
This paper presents a novel speaking-style captioning method that generates diverse descriptions while accurately predicting speaking-style information. Conventional learning criteria directly use original captions that contain not only speaking-style factor terms but also syntax words, which disturbs learning speaking-style information. To solve this problem, we introduce factor-conditioned captioning (FCC), which first outputs a phrase representing speaking-style factors (e.g., gender, pitch, etc.), and then generates a caption to ensure the model explicitly learns speaking-style factors. We also propose greedy-then-sampling (GtS) decoding, which first predicts speaking-style factors deterministically to guarantee semantic accuracy, and then generates a caption based on factor-conditioned sampling to ensure diversity. Experiments show that FCC outperforms the original caption-based training, and with GtS, it generates more diverse captions while keeping style prediction performance.
Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 27 Jun 2024.
Builds on
26 of 33 references.
References · 26
- 2023GPT-4 Technical Reportcited by 18,396
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 4,057
- 2019The Curious Case of Neural Text Degenerationcited by 2,916
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,342
- 2021WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processingcited by 1,657
- 2019LibriTTS: A Corpus Derived from LibriSpeech for Text-to-Speechcited by 649
- 2023Qwen-Audio: Advancing Universal Audio Understanding via Unified Large-Scale Audio-Language Modelscited by 608
14 more
- 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
- 2023Pengi: An Audio Language Model for Audio Taskscited by 220
- 2022PromptTTS: Controllable Text-to-Speech with Text Descriptionscited by 110
- 2023Joint Audio and Speech Understandingcited by 100
- 2023VioLA: Unified Codec Language Models for Speech Recognition, Synthesis, and Translationcited by 87
- 2023Whisper-AT: Noise-Robust Automatic Speech Recognizers are Also Strong General Audio Event Taggerscited by 72
- 2023SLM: Bridge the thin gap between speech and text foundation modelscited by 67
- 2023TextrolSpeech: A Text Style Control Speech Corpus With Codec Language Text-to-Speech Modelscited by 48
- 2024Advancing Large Language Models to Capture Varied Speaking Styles and Respond Properly in Spoken Conversationscited by 45
- 2023SECap: Speech Emotion Captioning with Large Language Modelcited by 33
- 2023PromptTTS++: Controlling Speaker Identity in Prompt-Based Text-to-Speech Using Natural Language Descriptionscited by 25
- 2022Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurementscited by 22
- 2023PromptVC: Flexible Stylistic Voice Conversion in Latent Space Driven by Natural Language Promptscited by 18
- 2023StyleCap: Automatic Speaking-Style Captioning from Speech Based on Speech and Language Self-supervised Learning Modelscited by 7
Built on by
Citing papers per year: 2025: 1; 2026: 1.
2 papers.