# Evaluating Prerequisite Qualities for Learning End-to-End Dialog Systems

arXiv: https://arxiv.org/abs/1511.06931 · HTML: https://mlsanity.com/paper/1511.06931/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2015.

## Built on by (most cited first, 30 of 34)

- [RedCaps: web-curated image-text data created by the people, for the people](https://mlsanity.com/paper/2111.11431/) (2021)
- [LM-Polygraph: Uncertainty Estimation for Language Models](https://mlsanity.com/paper/2311.07383/) (2023)
- [Advances and Challenges in Conversational Recommender Systems: A Survey](https://mlsanity.com/paper/2101.09459/) (2021)
- [Demystifying Chains, Trees, and Graphs of Thoughts](https://mlsanity.com/paper/2401.14295/) (2024)
- [RevCore: Review-augmented Conversational Recommendation](https://mlsanity.com/paper/2106.00957/) (2021)
- [DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation](https://mlsanity.com/paper/2109.08877/) (2021)
- [Assessing Hidden Risks of LLMs: An Empirical Study on Robustness, Consistency, and Credibility](https://mlsanity.com/paper/2305.10235/) (2023)
- [OpenViDial 2.0: A Larger-Scale, Open-Domain Dialogue Generation Dataset with Visual Contexts](https://mlsanity.com/paper/2109.12761/) (2021)
- [Data Augmentation with Paraphrase Generation and Entity Extraction for Multimodal Dialogue System](https://mlsanity.com/paper/2205.04006/) (2022)
- [MVP: Multi-task Supervised Pre-training for Natural Language Generation](https://mlsanity.com/paper/2206.12131/) (2022)
- [Social Influence Dialogue Systems: A Survey of Datasets and Models For Social Influence Tasks](https://mlsanity.com/paper/2210.05664/) (2022)
- [U-NEED: A Fine-grained Dataset for User Needs-Centric E-commerce Conversational Recommendation](https://mlsanity.com/paper/2305.04774/) (2023)
- [Talk the Walk: Synthetic Data Generation for Conversational Music Recommendation](https://mlsanity.com/paper/2301.11489/) (2023)
- [From Benchmarks to Skills: Low-Rank Factors for LLM Evaluation](https://mlsanity.com/paper/2507.20208/) (2025)
- [A Cooperative Memory Network for Personalized Task-oriented Dialogue Systems with Incomplete User Profiles](https://mlsanity.com/paper/2102.08322/) (2021)
- [Beyond Single Items: Exploring User Preferences in Item Sets with the Conversational Playlist Curation Dataset](https://mlsanity.com/paper/2303.06791/) (2023)
- [Knowledge Grounded Conversational Symptom Detection with Graph Memory Networks](https://mlsanity.com/paper/2101.09773/) (2021)
- [Modeling Performance in Open-Domain Dialogue with PARADISE](https://mlsanity.com/paper/2110.11164/) (2021)
- [A Survey of Personality, Persona, and Profile in Conversational Agents and Chatbots](https://mlsanity.com/paper/2401.00609/) (2023)
- [Artefact Retrieval: Overview of NLP Models with Knowledge Base Access](https://mlsanity.com/paper/2201.09651/) (2022)
- [Improving Open-Domain Dialogue Evaluation with a Causal Inference Model](https://mlsanity.com/paper/2301.13372/) (2023)
- [Measuring Aleatoric and Epistemic Uncertainty in LLMs: Empirical Evaluation on ID and OOD QA Tasks](https://mlsanity.com/paper/2511.03166/) (2025)
- [Pan More Gold from the Sand: Refining Open-domain Dialogue Training with Noisy Self-Retrieval Generation](https://mlsanity.com/paper/2201.11367/) (2022)
- [NLU for Game-based Learning in Real: Initial Evaluations](https://mlsanity.com/paper/2205.13754/) (2022)
- [End-to-End Evaluation of a Spoken Dialogue System for Learning Basic Mathematics](https://mlsanity.com/paper/2211.03511/) (2022)
- [A2P-MANN: Adaptive Attention Inference Hops Pruned Memory-Augmented Neural Networks](https://mlsanity.com/paper/2101.09693/) (2021)
- [SmartProbe: A Virtual Moderator for Market Research Surveys](https://mlsanity.com/paper/2305.08271/) (2023)
- [LASER: Stratified Selective Sampling for Instruction Tuning with Dedicated Scoring Strategy](https://mlsanity.com/paper/2505.22157/) (2025)
- [Synthetic Dialogue Generation for Interactive Conversational Elicitation & Recommendation (ICER)](https://mlsanity.com/paper/2510.02331/) (2025)
- [Avatar-Forever: Decoupled Parallel Training for High-Quality Real-Time Infinite Avatars](https://mlsanity.com/paper/2608.12107/) (2026)

## Abstract (arXiv)

A long-term goal of machine learning is to build intelligent conversational agents. One recent popular approach is to train end-to-end models on a large amount of real dialog transcripts between humans (Sordoni et al., 2015; Vinyals & Le, 2015; Shang et al., 2015). However, this approach leaves many questions unanswered as an understanding of the precise successes and shortcomings of each model is hard to assess. A contrasting recent proposal are the bAbI tasks (Weston et al., 2015b) which are synthetic data that measure the ability of learning machines at various reasoning tasks over toy language. Unfortunately, those tests are very small and hence may encourage methods that do not scale. In this work, we propose a suite of new tasks of a much larger scale that attempt to bridge the gap between the two regimes. Choosing the domain of movies, we provide tasks that test the ability of models to answer factual questions (utilizing OMDB), provide personalization (utilizing MovieLens), carry short conversations about the two, and finally to perform on natural dialogs from Reddit. We provide a dataset covering 75k movie entities and with 3.5M training examples. We present results of various models on these tasks, and evaluate their performance.