# TalkPlayData 2: An Agentic Synthetic Data Pipeline for Multimodal Conversational Music Recommendation

arXiv: https://arxiv.org/abs/2509.09685 · HTML: https://mlsanity.com/paper/2509.09685/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal and embodied agents, one of 16 directions in LLM agents and tool use.
- Builds on 27 of its 53 references that are in the map.
- First posted on arXiv 18 Aug 2025.

## Placed in

- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)
- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 27 of 27)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Robust Speech Recognition via Large-Scale Weak Supervision](https://mlsanity.com/paper/2212.04356/) (2022)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge](https://mlsanity.com/paper/2411.16594/) (2024)
- [MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark](https://mlsanity.com/paper/2402.04788/) (2024)
- [Self-Preference Bias in LLM-as-a-Judge](https://mlsanity.com/paper/2410.21819/) (2024)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [MMAR: A Challenging Benchmark for Deep Reasoning in Speech, Audio, Music, and Their Mix](https://mlsanity.com/paper/2505.13032/) (2025)
- [Conversational Recommender System](https://mlsanity.com/paper/1806.03277/) (2018)
- [madmom: a new Python Audio and Music Signal Processing Library](https://mlsanity.com/paper/1605.07008/) (2016)
- [LLM-Powered User Simulator for Recommender System](https://mlsanity.com/paper/2412.16984/) (2024)
- [LLark: A Multimodal Instruction-Following Language Model for Music](https://mlsanity.com/paper/2310.07160/) (2023)
- [Do We Truly Need So Many Samples? Multi-LLM Repeated Sampling Efficiently Scales Test-Time Compute](https://mlsanity.com/paper/2504.00762/) (2025)
- [Content-driven Music Recommendation: Evolution, State of the Art, and Challenges](https://mlsanity.com/paper/2107.11803/) (2021)
- [Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation](https://mlsanity.com/paper/2301.11489/) (2023)
- [The Music Maestro or The Musically Challenged, A Massive Music Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2406.15885/) (2024)
- [Incorporating External Knowledge and Goal Guidance for LLM-based Conversational Recommender Systems](https://mlsanity.com/paper/2405.01868/) (2024)
- [Beyond Single Items: Exploring User Preferences in Item Sets with the Conversational Playlist Curation Dataset](https://mlsanity.com/paper/2303.06791/) (2023)
- [Can LLMs "Reason" in Music? An Evaluation of LLMs' Capability of Music Understanding and Generation](https://mlsanity.com/paper/2407.21531/) (2024)
- [TALKPLAY: Multimodal Music Recommendation with Large Language Models](https://mlsanity.com/paper/2502.13713/) (2025)
- [Just Ask for Music (JAM): Multimodal and Personalized Natural Language Music Recommendation](https://mlsanity.com/paper/2507.15826/) (2025)
- [Evaluation of pretrained language models on music understanding](https://mlsanity.com/paper/2409.11449/) (2024)
- [Music Discovery Dialogue Generation Using Human Intent Analysis and Large Language Models](https://mlsanity.com/paper/2411.07439/) (2024)
- [A Benchmark and Robustness Study of In-Context-Learning with Large Language Models in Music Entity Detection](https://mlsanity.com/paper/2412.11851/) (2024)
- [SampleLLM: Optimizing Tabular Data Synthesis in Recommendations](https://mlsanity.com/paper/2501.16125/) (2025)
- [AgentSGEN: Multi-Agent LLM in the Loop for Semantic Collaboration and GENeration of Synthetic Data](https://mlsanity.com/paper/2505.13466/) (2025)
- [Predicting User Intents and Musical Attributes from Music Discovery Conversations](https://mlsanity.com/paper/2411.12254/) (2024)

## Built on by (most cited first, 3 of 3)

- [Autonomous Information Seeking: A Roadmap for Agentic Recommender Systems](https://mlsanity.com/paper/2607.04433/) (2026)
- [Music Recommendation with Large Language Models: Challenges, Opportunities, and Evaluation](https://mlsanity.com/paper/2511.16478/) (2025)
- [Bootstrapping Conversational Recommendation Agents At Spotify: Synthetic Data Generation and Self-Improvement Loops](https://mlsanity.com/paper/2609.30297/) (2026)

## Abstract (arXiv)

We present TalkPlayData 2, a synthetic dataset for multimodal conversational music recommendation generated by an agentic data pipeline. In the proposed pipeline, multiple large language model (LLM) agents are created under various roles with specialized prompts and access to different parts of information, and the chat data is acquired by logging the conversation between the Listener LLM and the Recsys LLM. To cover various conversation scenarios, for each conversation, the Listener LLM is conditioned on a finetuned conversation goal. Finally, all the LLMs are multimodal with audio and images, allowing a simulation of multimodal recommendation and conversation. In the LLM-as-a-judge and subjective evaluation experiments, TalkPlayData 2 achieved the proposed goal in various aspects related to training a generative recommendation model for music. TalkPlayData 2 and its generation code are released at https://talkpl-ai.github.io.