mlsanityData to 7 Oct 2026

Evaluation of Synthetic Datasets for Conversational Recommender Systems

arXiv 2212.08167 ↗First posted 12 Dec 2022cs.CL cs.AI cs.IRcited by 6

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

For researchers leveraging Large-Language Models (LLMs) in the generation of training datasets, especially for conversational recommender systems - the absence of robust evaluation frameworks has been a long-standing problem. The efficiency brought about by LLMs in the data generation phase is impeded during the process of evaluation of the generated data, since it generally requires human-raters to ensure that the data generated is of high quality and has sufficient diversity. Since the quality of training data is critical for downstream applications, it is important to develop metrics that evaluate the quality holistically and identify biases. In this paper, we present a framework that takes a multi-faceted approach towards evaluating datasets produced by generative models and discuss the advantages and limitations of various evaluation methods.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 9 of its 11 references that are in the map. First posted on arXiv 12 Dec 2022.

Builds on

9 of 11 references.

References · 9

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  3. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  4. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  5. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  6. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  7. 2020A Survey on Conversational Recommender Systemscited by 142
  8. 2019Unifying Human and Statistical Evaluation for Natural Language Generationcited by 127
  9. 2017Composite Task-Completion Dialogue Policy Learning via Hierarchical Deep Reinforcement Learningcited by 44

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 1.

6 papers.

  1. 2023Talk the Walk: Synthetic Data Generation for Conversational Music Recommendationcited by 9
  2. 2024Automatic Histograms: Leveraging Language Models for Text Dataset Explorationcited by 8
  3. 2024Exploring Empty Spaces: Human-in-the-Loop Data Augmentationcited by 6
  4. 2024The Evolution of LLM Adoption in Industry Data Curation Practicescited by 6
  5. 2023Visualizing Linguistic Diversity of Text Datasets Synthesized by Large Language Modelscited by 4
  6. 2025Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)cited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Principled Synthetic Data Enables the First Scaling Laws for LLMs in Recommendation
  2. 2023Rethinking the Evaluation for Conversational Recommendation in the Era of Large Language Models
  3. 2026A Survey on Evaluating Quality and Trustworthiness in LLM-Generated Data
  4. 2024Beyond Retrieval: Generating Narratives in Conversational Recommender Systems
  5. 2025Evaluating Conversational Recommender Systems via Large Language Models: A User-Centric Framework
  6. 2025Generative Large Recommendation Models: Emerging Trends in LLMs for Recommendation
  7. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  8. 2025GR-LLMs: Recent Advances in Generative Recommendation Based on Large Language Models