# WinSyn: An Automated Pipeline for Realistic Enterprise Question-Answering Evaluation

arXiv: https://arxiv.org/abs/2609.12171 · HTML: https://mlsanity.com/paper/2609.12171/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Agentic RAG, agents and memory, one of 15 directions in Retrieval-augmented generation.
- Builds on 11 of its 32 references that are in the map.
- First posted on arXiv 10 Sep 2026.

## Placed in

- Retrieval-augmented generation / [Agentic RAG, agents and memory](https://mlsanity.com/rag/agentic-rag-agents-and-memory/)

## Builds on (most cited first, 11 of 11)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks](https://mlsanity.com/paper/2412.14161/) (2024)
- [DeepSearchQA: Bridging the Comprehensiveness Gap for Deep Research Agents](https://mlsanity.com/paper/2601.20975/) (2026)
- [CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions](https://mlsanity.com/paper/2505.18878/) (2025)
- [OfficeQA Pro: An Enterprise Benchmark for End-to-End Grounded Reasoning](https://mlsanity.com/paper/2603.08655/) (2026)
- [PRBench: Large-Scale Expert Rubrics for Evaluating High-Stakes Professional Reasoning](https://mlsanity.com/paper/2511.11562/) (2025)
- [Finding Flawed Fictions: Evaluating Complex Reasoning in Language Models via Plot Hole Detection](https://mlsanity.com/paper/2504.11900/) (2025)
- [Benchmarking Deep Search over Heterogeneous Enterprise Data](https://mlsanity.com/paper/2506.23139/) (2025)
- [DRBench: A Realistic Benchmark for Enterprise Deep Research](https://mlsanity.com/paper/2510.00172/) (2025)
- [DRACO: a Cross-Domain Benchmark for Deep Research Accuracy, Completeness, and Objectivity](https://mlsanity.com/paper/2602.11685/) (2026)

## Abstract (arXiv)

Enterprise settings provide a challenging environment for question-answering agents, which often rely on Retrieval-Augmented Generation, Deep Research (DR), and related techniques. Much of this challenge comes from the complexity of enterprise data: information is often spread across evolving and potentially conflict- ing emails, chat messages, documents, and other artifacts. Existing benchmarks typically have limited real-world complexity, short-form responses, and unnatural queries, so they often fail to capture the challenges of enterprise settings. In this work, we introduce an automated pipeline for generating synthetic datasets of emails reflecting realistic workplace scenarios, along with long- and short-form questions and gold answers grounded in the data. Our method simulates long-running enterprise projects spanning several months and involving up to 25 interacting employees across multiple roles. The data emphasizes ambiguity, distributed information, and naturally occurring queries. To validate the pipeline, we evaluate few standard agentic baselines on our datasets using the latest frontier models. We find that aggregate scores averaged over all queries remain below 80% for each dataset, indicating significant room for improvement. These findings suggest that more work remains to be done for enterprise deployment and underscore the importance of realistic, high-complexity evaluation data for developing stronger real-world enterprise DR systems.