# Are Deep Neural Networks SMARTer than Second Graders?

arXiv: https://arxiv.org/abs/2212.09993 · HTML: https://mlsanity.com/paper/2212.09993/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 49 of its 87 references that are in the map.
- First posted on arXiv 20 Dec 2022.

## Builds on (most cited first, 30 of 49)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [FLAVA: A Foundational Language And Vision Alignment Model](https://mlsanity.com/paper/2112.04482/) (2021)
- [From Recognition to Cognition: Visual Commonsense Reasoning](https://mlsanity.com/paper/1811.10830/) (2018)
- [Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionality](https://mlsanity.com/paper/2204.03162/) (2022)
- [The Hateful Memes Challenge: Detecting Hate Speech in Multimodal Memes](https://mlsanity.com/paper/2005.04790/) (2020)
- [A Corpus for Reasoning About Natural Language Grounded in Photographs](https://mlsanity.com/paper/1811.00491/) (2018)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [RAVEN: A Dataset for Relational and Analogical Visual rEasoNing](https://mlsanity.com/paper/1903.02741/) (2019)
- [Measuring abstract reasoning in neural networks](https://mlsanity.com/paper/1807.04225/) (2018)
- [Grad-CAM: Why did you say that?](https://mlsanity.com/paper/1611.07450/) (2016)
- [DreamCoder: Growing generalizable, interpretable knowledge with wake-sleep Bayesian program learning](https://mlsanity.com/paper/2006.08381/) (2020)
- [CrossFormer: A Versatile Vision Transformer Hinging on Cross-scale Attention](https://mlsanity.com/paper/2108.00154/) (2021)
- [Learning by Abstraction: The Neural State Machine](https://mlsanity.com/paper/1907.03950/) (2019)
- [Explainable and Explicit Visual Reasoning over Scene Graphs](https://mlsanity.com/paper/1812.01855/) (2018)
- [Abstraction and Analogy-Making in Artificial Intelligence](https://mlsanity.com/paper/2102.10717/) (2021)
- [Benchmarking Neural Network Robustness to Common Corruptions and Surface Variations](https://mlsanity.com/paper/1807.01697/) (2018)

## Built on by (most cited first, 24 of 24)

- [DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understanding](https://mlsanity.com/paper/2412.10302/) (2024)
- [One Small Step for Generative AI, One Giant Leap for AGI: A Complete Survey on ChatGPT in AIGC Era](https://mlsanity.com/paper/2304.06488/) (2023)
- [VisualPuzzles: Decoupling Multimodal Reasoning Evaluation from Domain Knowledge](https://mlsanity.com/paper/2504.10342/) (2025)
- [ChatGPT and a New Academic Reality: Artificial Intelligence-Written Research Papers and the Ethics of the Large Language Models in Scholarly Publishing](https://mlsanity.com/paper/2303.13367/) (2023)
- [Foundation Models in Autonomous Driving: A Survey on Scenario Generation and Scenario Analysis](https://mlsanity.com/paper/2506.11526/) (2025)
- [KiVA: Kid-inspired Visual Analogies for Testing Large Multimodal Models](https://mlsanity.com/paper/2407.17773/) (2024)
- [Evaluating Large Vision-and-Language Models on Children's Mathematical Olympiads](https://mlsanity.com/paper/2406.15736/) (2024)
- [AI, write an essay for me: A large-scale comparison of human-written versus ChatGPT-generated essays](https://mlsanity.com/paper/2304.14276/) (2023)
- [Towards Learning Abductive Reasoning using VSA Distributed Representations](https://mlsanity.com/paper/2406.19121/) (2024)
- [Megrez-Omni Technical Report](https://mlsanity.com/paper/2502.15803/) (2025)
- [Smart Vision-Language Reasoners](https://mlsanity.com/paper/2407.04212/) (2024)
- [Can Large Reasoning Models do Analogical Reasoning under Perceptual Uncertainty?](https://mlsanity.com/paper/2503.11207/) (2025)
- [TurtleBench: A Visual Programming Benchmark in Turtle Geometry](https://mlsanity.com/paper/2411.00264/) (2024)
- [Task-Core Memory Management and Consolidation for Long-term Continual Learning](https://mlsanity.com/paper/2505.09952/) (2025)
- [I-RAVEN-X: Benchmarking Generalization and Robustness of Analogical and Mathematical Reasoning in Large Language and Reasoning Models](https://mlsanity.com/paper/2510.17496/) (2025)
- [Can AI Understand the Language of Origami?](https://mlsanity.com/paper/2603.13856/) (2026)
- [ChemLabs on ChemO: A Multi-Agent System for Multimodal Reasoning on IChO 2025](https://mlsanity.com/paper/2511.16205/) (2025)
- [Solution for SMART-101 Challenge of ICCV Multi-modal Algorithmic Reasoning Task 2023](https://mlsanity.com/paper/2310.06440/) (2023)
- [What If the TV Was Off? Examining Counterfactual Reasoning Abilities of Multi-modal Language Models](https://mlsanity.com/paper/2310.06627/) (2023)
- [Integrating Text and Image Pre-training for Multi-modal Algorithmic Reasoning](https://mlsanity.com/paper/2406.05318/) (2024)
- [Solution for SMART-101 Challenge of CVPR Multi-modal Algorithmic Reasoning Task 2024](https://mlsanity.com/paper/2406.05963/) (2024)
- [Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation](https://mlsanity.com/paper/2410.18001/) (2024)
- [Action Without Interaction: Probing the Physical Foundations of Video LMMs via Contact-Release Detection](https://mlsanity.com/paper/2511.20162/) (2025)
- [WISE: Weighted Iterative Society-of-Experts for Robust Multimodal Multi-Agent Debate](https://mlsanity.com/paper/2512.02405/) (2025)

## Abstract (arXiv)

Recent times have witnessed an increasing number of applications of deep neural networks towards solving tasks that require superior cognitive abilities, e.g., playing Go, generating art, ChatGPT, etc. Such a dramatic progress raises the question: how generalizable are neural networks in solving problems that demand broad skills? To answer this question, we propose SMART: a Simple Multimodal Algorithmic Reasoning Task and the associated SMART-101 dataset, for evaluating the abstraction, deduction, and generalization abilities of neural networks in solving visuo-linguistic puzzles designed specifically for children in the 6--8 age group. Our dataset consists of 101 unique puzzles; each puzzle comprises a picture and a question, and their solution needs a mix of several elementary skills, including arithmetic, algebra, and spatial reasoning, among others. To scale our dataset towards training deep neural networks, we programmatically generate entirely new instances for each puzzle, while retaining their solution algorithm. To benchmark performances on SMART-101, we propose a vision and language meta-learning model using varied state-of-the-art backbones. Our experiments reveal that while powerful deep models offer reasonable performances on puzzles in a supervised setting, they are not better than random accuracy when analyzed for generalization. We also evaluate the recent ChatGPT and other large language models on a subset of SMART-101 and find that while these models show convincing reasoning abilities, the answers are often incorrect.