# Evaluation of Synthetic Datasets for Conversational Recommender Systems

arXiv: https://arxiv.org/abs/2212.08167 · HTML: https://mlsanity.com/paper/2212.08167/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 9 of its 11 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Placed in

- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 9 of 9)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Text](https://mlsanity.com/paper/2107.00061/) (2021)
- [Data Augmentation using Pre-trained Transformer Models](https://mlsanity.com/paper/2003.02245/) (2020)
- [Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Text](https://mlsanity.com/paper/2202.06935/) (2022)
- [A Survey on Conversational Recommender Systems](https://mlsanity.com/paper/2004.00646/) (2020)
- [Unifying Human and Statistical Evaluation for Natural Language Generation](https://mlsanity.com/paper/1904.02792/) (2019)
- [Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learning](https://mlsanity.com/paper/1704.03084/) (2017)

## Built on by (most cited first, 6 of 6)

- [Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation](https://mlsanity.com/paper/2301.11489/) (2023)
- [Automatic Histograms: Leveraging Language Models for Text Dataset Exploration](https://mlsanity.com/paper/2402.14880/) (2024)
- [Exploring Empty Spaces: Human-in-the-Loop Data Augmentation](https://mlsanity.com/paper/2410.01088/) (2024)
- [The Evolution of LLM Adoption in Industry Data Curation Practices](https://mlsanity.com/paper/2412.16089/) (2024)
- [Visualizing Linguistic Diversity of Text Datasets Synthesized by Large Language Models](https://mlsanity.com/paper/2305.11364/) (2023)
- [Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)](https://mlsanity.com/paper/2510.02331/) (2025)

## Abstract (arXiv)

For researchers leveraging Large-Language Models (LLMs) in the generation of training datasets, especially for conversational recommender systems - the absence of robust evaluation frameworks has been a long-standing problem. The efficiency brought about by LLMs in the data generation phase is impeded during the process of evaluation of the generated data, since it generally requires human-raters to ensure that the data generated is of high quality and has sufficient diversity. Since the quality of training data is critical for downstream applications, it is important to develop metrics that evaluate the quality holistically and identify biases. In this paper, we present a framework that takes a multi-faceted approach towards evaluating datasets produced by generative models and discuss the advantages and limitations of various evaluation methods.