mlsanityData to 7 Oct 2026

MIMDE: Exploring the Use of Synthetic vs Human Data for Evaluating Multi-Insight Multi-Document Extraction Tasks

arXiv 2411.19689 ↗First posted 29 Nov 2024cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities in text analysis tasks, yet their evaluation on complex, real-world applications remains challenging. We define a set of tasks, Multi-Insight Multi-Document Extraction (MIMDE) tasks, which involves extracting an optimal set of insights from a document corpus and mapping these insights back to their source documents. This task is fundamental to many practical applications, from analyzing survey responses to processing medical records, where identifying and tracing key insights across documents is crucial. We develop an evaluation framework for MIMDE and introduce a novel set of complementary human and synthetic datasets to examine the potential of synthetic data for LLM evaluation. After establishing optimal metrics for comparing extracted insights, we benchmark 20 state-of-the-art LLMs on both datasets. Our analysis reveals a strong correlation (0.71) between the ability of LLMs to extracts insights on our two datasets but synthetic data fails to capture the complexity of document-level analysis. These findings offer crucial guidance for the use of synthetic data in evaluating text analysis systems, highlighting both its potential and limitations.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 53 references that are in the map. First posted on arXiv 29 Nov 2024.

Builds on

21 of 53 references.

References · 21

  1. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  2. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 732
  4. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 397
  5. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  6. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 262
  7. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  8. 2019Taskmaster-1: Toward a Realistic and Diverse Dialog Datasetcited by 139
  9. 2024Large Language Models Meet NLP: A Surveycited by 102
  10. 2023Large Language Models Enable Few-Shot Clusteringcited by 72
  11. 2023LLM-in-the-loop: Leveraging Large Language Model for Thematic Analysiscited by 70
  12. 2022Synthetic Data -- what, why and how?cited by 66
9 more
  1. 2022Structured information extraction from complex scientific text with fine-tuned large language modelscited by 56
  2. 2023Summarization is (Almost) Deadcited by 43
  3. 2024Synthetic Dialogue Dataset Generation using LLM Agentscited by 39
  4. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  5. 2018Zero-Shot Dialog Generation with Cross-Domain Latent Actionscited by 23
  6. 2021Do Question Answering Modeling Improvements Hold Across Benchmarks?cited by 20
  7. 2024Text Clustering as Classification with LLMscited by 16
  8. 2023Text Encoders Lack Knowledge: Leveraging Generative LLMs for Domain-Specific Semantic Textual Similaritycited by 11
  9. 2024Exploring the Human-LLM Synergy in Advancing Theory-driven Qualitative Analysiscited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  2. 2025Grounding Synthetic Data Evaluations of Language Models in Unsupervised Document Corpora
  3. 2024Under the Surface: Tracking the Artifactuality of LLM-Generated Data
  4. 2026LLM as a Meta-Judge: Synthetic Data for NLP Evaluation Metric Validation
  5. 2024Efficacy of Synthetic Data as a Benchmark
  6. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation
  7. 2024A Review on Scientific Knowledge Extraction using Large Language Models in Biomedical Sciences
  8. 2025Scaling Laws of Synthetic Data for Language Models