# DiNeR: a Large Realistic Dataset for Evaluating Compositional Generalization

arXiv: https://arxiv.org/abs/2406.04669 · HTML: https://mlsanity.com/paper/2406.04669/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 34 references that are in the map.
- First posted on arXiv 7 Jun 2024.

## Builds on (most cited first, 20 of 20)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distribution](https://mlsanity.com/paper/2202.10054/) (2022)
- [Measuring Compositional Generalization: A Comprehensive Method on Realistic Data](https://mlsanity.com/paper/1912.09713/) (2019)
- [COGS: A Compositional Generalization Challenge Based on Semantic Interpretation](https://mlsanity.com/paper/2010.05465/) (2020)
- [Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?](https://mlsanity.com/paper/2010.12725/) (2020)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Diverse Demonstrations Improve In-context Compositional Generalization](https://mlsanity.com/paper/2212.06800/) (2022)
- [Compositional generalization through meta sequence-to-sequence learning](https://mlsanity.com/paper/1906.05381/) (2019)
- [Learning Compositional Rules via Neural Program Synthesis](https://mlsanity.com/paper/2003.05562/) (2020)
- [FLEX: Unifying Evaluation for Few-Shot NLP](https://mlsanity.com/paper/2107.07170/) (2021)
- [A Broad Study of Pre-training for Domain Generalization and Adaptation](https://mlsanity.com/paper/2203.11819/) (2022)
- [How Do In-Context Examples Affect Compositional Generalization?](https://mlsanity.com/paper/2305.04835/) (2023)
- [Evaluating the Impact of Model Scale for Compositional Generalization in Semantic Parsing](https://mlsanity.com/paper/2205.12253/) (2022)
- [Human few-shot learning of compositional instructions](https://mlsanity.com/paper/1901.04587/) (2019)
- [Mengzi: Towards Lightweight yet Ingenious Pre-trained Models for Chinese](https://mlsanity.com/paper/2110.06696/) (2021)
- [Jump to better conclusions: SCAN both left and right](https://mlsanity.com/paper/1809.04640/) (2018)
- [On the Compositional Generalization Gap of In-Context Learning](https://mlsanity.com/paper/2211.08473/) (2022)
- [Domain-Adaptive Pretraining Methods for Dialogue Understanding](https://mlsanity.com/paper/2105.13665/) (2021)
- [Learning to Perform Complex Tasks through Compositional Fine-Tuning of Language Models](https://mlsanity.com/paper/2210.12607/) (2022)
- [Counterfactual Recipe Generation: Exploring Compositional Generalization in a Realistic Scenario](https://mlsanity.com/paper/2210.11431/) (2022)

## Built on by (most cited first, 1 of 1)

- [Evaluating and Improving Robustness in Large Language Models: A Survey and Future Directions](https://mlsanity.com/paper/2506.11111/) (2025)

## Abstract (arXiv)

Most of the existing compositional generalization datasets are synthetically-generated, resulting in a lack of natural language variation. While there have been recent attempts to introduce non-synthetic datasets for compositional generalization, they suffer from either limited data scale or a lack of diversity in the forms of combinations. To better investigate compositional generalization with more linguistic phenomena and compositional diversity, we propose the DIsh NamE Recognition (DiNeR) task and create a large realistic Chinese dataset. Given a recipe instruction, models are required to recognize the dish name composed of diverse combinations of food, actions, and flavors. Our dataset consists of 3,811 dishes and 228,114 recipes, and involves plenty of linguistic phenomena such as anaphora, omission and ambiguity. We provide two strong baselines based on T5 and large language models (LLMs). This work contributes a challenging task, baseline methods to tackle the task, and insights into compositional generalization in the context of dish name recognition. Code and data are available at https://github.com/Jumpy-pku/DiNeR.