# CURI: A Benchmark for Productive Concept Learning Under Uncertainty

arXiv: https://arxiv.org/abs/2010.02855 · HTML: https://mlsanity.com/paper/2010.02855/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 6 Oct 2020.

## Built on by (most cited first, 21 of 21)

- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [Explain Before You Answer: A Survey on Compositional Visual Reasoning](https://mlsanity.com/paper/2508.17298/) (2025)
- [CompoSuite: A Compositional Reinforcement Learning Benchmark](https://mlsanity.com/paper/2207.04136/) (2022)
- [ReaSCAN: Compositional Reasoning in Language Grounding](https://mlsanity.com/paper/2109.08994/) (2021)
- [Pix2Code: Learning to Compose Neural Visual Concepts as Programs](https://mlsanity.com/paper/2402.08280/) (2024)
- [MEWL: Few-shot multimodal word learning with referential uncertainty](https://mlsanity.com/paper/2306.00503/) (2023)
- [CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanations](https://mlsanity.com/paper/2204.02380/) (2022)
- [Bongard in Wonderland: Visual Puzzles that Still Make AI Go Mad?](https://mlsanity.com/paper/2410.19546/) (2024)
- [Neural Concept Binder](https://mlsanity.com/paper/2406.09949/) (2024)
- [Interactive Visual Reasoning under Uncertainty](https://mlsanity.com/paper/2206.09203/) (2022)
- [V-LoL: A Diagnostic Dataset for Visual Logical Learning](https://mlsanity.com/paper/2306.07743/) (2023)
- [UBench: Benchmarking Uncertainty in Large Language Models with Multiple Choice Questions](https://mlsanity.com/paper/2406.12784/) (2024)
- [A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics](https://mlsanity.com/paper/2103.01403/) (2021)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence](https://mlsanity.com/paper/2103.00519/) (2021)
- [Reference-Limited Compositional Zero-Shot Learning](https://mlsanity.com/paper/2208.10046/) (2022)
- [Does MAML Only Work via Feature Re-use? A Data Centric Perspective](https://mlsanity.com/paper/2112.13137/) (2021)
- [A Data Source for Reasoning Embodied Agents](https://mlsanity.com/paper/2309.07974/) (2023)
- [A Concept-Centric Approach to Multi-Modality Learning](https://mlsanity.com/paper/2412.13847/) (2024)
- [Navigating High Dimensional Concept Space with Metalearning](https://mlsanity.com/paper/2508.01948/) (2025)
- [COCOLogic-V2: Identifying Logical Inconsistencies via Truly Hard-Negatives](https://mlsanity.com/paper/2606.28194/) (2026)

## Abstract (arXiv)

Humans can learn and reason under substantial uncertainty in a space of infinitely many concepts, including structured relational concepts ("a scene with objects that have the same color") and ad-hoc categories defined through goals ("objects that could fall on one's head"). In contrast, standard classification benchmarks: 1) consider only a fixed set of category labels, 2) do not evaluate compositional concept learning and 3) do not explicitly capture a notion of reasoning under uncertainty. We introduce a new few-shot, meta-learning benchmark, Compositional Reasoning Under Uncertainty (CURI) to bridge this gap. CURI evaluates different aspects of productive and systematic generalization, including abstract understandings of disentangling, productive generalization, learning boolean operations, variable binding, etc. Importantly, it also defines a model-independent "compositionality gap" to evaluate the difficulty of generalizing out-of-distribution along each of these axes. Extensive evaluations across a range of modeling choices spanning different modalities (image, schemas, and sounds), splits, privileged auxiliary concept information, and choices of negatives reveal substantial scope for modeling advances on the proposed task. All code and datasets will be available online.