mlsanityData to 7 Oct 2026

A Minimalist Dataset for Systematic Generalization of Perception, Syntax, and Semantics

arXiv 2103.01403 ↗First posted 2 Mar 2021cs.LG cs.AI cs.CVcited by 6

Not on the map. Why?

Abstract (arXiv)

Inspired by humans' exceptional ability to master arithmetic and generalize to new problems, we present a new dataset, Handwritten arithmetic with INTegers (HINT), to examine machines' capability of learning generalizable concepts at three levels: perception, syntax, and semantics. In HINT, machines are tasked with learning how concepts are perceived from raw signals such as images (i.e., perception), how multiple concepts are structurally combined to form a valid expression (i.e., syntax), and how concepts are realized to afford various reasoning tasks (i.e., semantics), all in a weakly supervised manner. Focusing on systematic generalization, we carefully design a five-fold test set to evaluate both the interpolation and the extrapolation of learned concepts w.r.t. the three levels. Further, we design a few-shot learning split to determine whether or not models can rapidly learn new concepts and generalize them to more complex scenarios. To comprehend existing models' limitations, we undertake extensive experiments with various sequence-to-sequence models, including RNNs, Transformers, and GPT-3 (with the chain of thought prompting). The results indicate that current models struggle to extrapolate to long-range syntactic dependency and semantics. Models exhibit a considerable gap toward human-level generalization when evaluated with new concepts in a few-shot setting. Moreover, we discover that it is infeasible to solve HINT by merely scaling up the dataset and the model size; this strategy contributes little to the extrapolation of syntax and semantics. Finally, in zero-shot GPT-3 experiments, the chain of thought prompting exhibits impressive results and significantly boosts the test accuracy. We believe the HINT dataset and the experimental findings are of great interest to the learning community on systematic generalization.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 59 of its 116 references that are in the map. First posted on arXiv 2 Mar 2021.

Builds on

59 of 116 references.

References · 59

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  10. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  11. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  12. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
38 more
  1. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  2. 2019SpecAugment: A Simple Data Augmentation Method for Automatic Speech Recognitioncited by 1,355
  3. 2018Self-Attention with Relative Position Representationscited by 1,112
  4. 2013Speech Recognition with Deep Recurrent Neural Networkscited by 666
  5. 2018Universal Transformerscited by 613
  6. 2020Scaling Laws for Autoregressive Generative Modelingcited by 510
  7. 2019Neural Probabilistic Logic Programming in DeepProbLogcited by 389
  8. 2019Analysing Mathematical Reasoning Abilities of Neural Modelscited by 331
  9. 2019Measuring Compositional Generalization: A Comprehensive Method on Realistic Datacited by 280
  10. 2019Compositionality decomposed: how do neural networks generalise?cited by 264
  11. 2020COGS: A Compositional Generalization Challenge Based on Semantic Interpretationcited by 260
  12. 2019RAVEN: A Dataset for Relational and Analogical Visual rEasoNingcited by 225
  13. 2017Learning Explanatory Rules from Noisy Datacited by 210
  14. 2018Constituency Parsing with a Self-Attentive Encodercited by 196
  15. 2018Measuring abstract reasoning in neural networkscited by 165
  16. 2020DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learningcited by 159
  17. 2019Good-Enough Compositional Data Augmentationcited by 158
  18. 2020Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both?cited by 147
  19. 2016DeepCoder: Learning to Write Programscited by 145
  20. 2017RobustFill: Neural Program Learning under Noisy I/Ocited by 124
  21. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  22. 2019Compositional generalization through meta sequence-to-sequence learningcited by 98
  23. 2020Compositional Generalization in Semantic Parsing: Pre-training vs. Specialized Architecturescited by 95
  24. 2020Compositional Generalization via Neural-Symbolic Stack Machinescited by 80
  25. 2020Learning Compositional Rules via Neural Program Synthesiscited by 74
  26. 2017Learning to Infer Graphics Programs from Hand-Drawn Imagescited by 73
  27. 2018Human-centric Indoor Scene Synthesis Using Stochastic Grammarcited by 73
  28. 2019Holistic++ Scene Understanding: Single-view 3D Holistic Scene Parsing and Human Pose Estimation with Human-Object Interaction and Physical Commonsensecited by 71
  29. 2018Holistic 3D Scene Parsing and Reconstruction from a Single RGB Imagecited by 69
  30. 2020Learning to Recombine and Resample Data for Compositional Generalizationcited by 65
  31. 2019CLOSURE: Assessing Systematic Generalization of CLEVR Modelscited by 63
  32. 2020Closed Loop Neural-Symbolic Learning via Integrating Neural Perception, Grammar Parsing, and Symbolic Reasoningcited by 58
  33. 2019Learning Perceptual Inference by Contrastingcited by 57
  34. 2019Compositional generalization in a deep seq2seq model by separating syntax and semanticscited by 56
  35. 2020Compositional Generalization by Learning Analytical Expressionscited by 53
  36. 2018Cooperative Holistic Scene Understanding: Unifying 3D Object, Layout, and Camera Pose Estimationcited by 51
  37. 2020LEMMA: A Multi-view Dataset for Learning Multi-agent Multi-task Activitiescited by 46
  38. 2020The EOS Decision and Length Extrapolationcited by 46

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 3; 2023: 3.

6 papers.

  1. 2023SparseDFF: Sparse-View Feature Distillation for One-Shot Dexterous Manipulationcited by 22
  2. 2023MEWL: Few-shot multimodal word learning with referential uncertaintycited by 17
  3. 2022Neural-Symbolic Recursive Machine for Systematic Generalizationcited by 12
  4. 2022Interactive Visual Reasoning under Uncertaintycited by 8
  5. 2023Active Reasoning in an Open-World Environmentcited by 8
  6. 2022On the Complexity of Bayesian Generalizationcited by 3