mlsanityData to 14 Oct 2026

The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision

arXiv 1904.12584 ↗First posted 26 Apr 2019cs.CV cs.AI cs.CLcited by 372

Not on the map. Why?

Cited by 74 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose the Neuro-Symbolic Concept Learner (NS-CL), a model that learns visual concepts, words, and semantic parsing of sentences without explicit supervision on any of them; instead, our model learns by simply looking at images and reading paired questions and answers. Our model builds an object-based scene representation and translates sentences into executable, symbolic programs. To bridge the learning of two modules, we use a neuro-symbolic reasoning module that executes these programs on the latent scene representation. Analogical to human concept learning, the perception module learns visual concepts based on the language description of the object being referred to. Meanwhile, the learned visual concepts facilitate learning new words and parsing new sentences. We use curriculum learning to guide the searching over the large compositional space of images and language. Extensive experiments demonstrate the accuracy and efficiency of our model on learning visual concepts, word representations, and semantic parsing of sentences. Further, our method allows easy generalization to new object attributes, compositions, language concepts, scenes and questions, and even new program domains. It also empowers applications including visual question answering and bidirectional image-text retrieval.

Key facts. Cited by 372 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Apr 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 65; 2022: 66; 2023: 77; 2024: 70; 2025: 51; 2026: 43.

372 papers.

Sort:

By area:

  1. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  2. 2023Logic-LM: Empowering Large Language Models with Symbolic Solvers for Faithful Logical Reasoningcited by 391
  3. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  4. 2024ReKep: Spatio-Temporal Reasoning of Relational Keypoint Constraints for Robotic Manipulationcited by 315
  5. 2021GeoQA: A Geometric Question Answering Benchmark Towards Multimodal Numerical Reasoningcited by 228
  6. 2021D'ya like DAGs? A Survey on Structure Learning and Causal Discoverycited by 211
  7. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  8. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  9. 2023Explainable Artificial Intelligence (XAI) 2.0: A Manifesto of Open Challenges and Interdisciplinary Research Directionscited by 125
  10. 2021AGQA: A Benchmark for Compositional Spatio-Temporal Reasoningcited by 120
  11. 2022CLEVR-Math: A Dataset for Compositional Language, Visual and Mathematical Reasoningcited by 120
  12. 2023Hypothesis Search: Inductive Reasoning with Language Modelscited by 120
38 more
  1. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  2. 2021Abstraction and Analogy-Making in Artificial Intelligencecited by 103
  3. 2024Towards Guaranteed Safe AI: A Framework for Ensuring Robust and Reliable AI Systemscited by 96
  4. 2022Super-CLEVR: A Virtual Benchmark to Diagnose Domain Robustness in Visual Reasoningcited by 92
  5. 2023NeurASP: Embracing Neural Networks into Answer Set Programmingcited by 92
  6. 2021Improving Coherence and Consistency in Neural Sequence Models with Dual-System, Neuro-Symbolic Reasoningcited by 85
  7. 2026PointWorld: Scaling 3D World Models for In-The-Wild Robotic Manipulationcited by 82
  8. 2021From Statistical Relational to Neurosymbolic Artificial Intelligence: a Surveycited by 72
  9. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  10. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 68
  11. 2022A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matricescited by 64
  12. 2023Scallop: A Language for Neurosymbolic Programmingcited by 63
  13. 2022Theory of Graph Neural Networks: Representation and Learningcited by 61
  14. 20233D Concept Learning and Reasoning from Multi-View Imagescited by 60
  15. 2023Large Language Models are Visual Reasoning Coordinatorscited by 58
  16. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  17. 2023Interpretable Neural-Symbolic Concept Reasoningcited by 57
  18. 2021Dynamic Visual Reasoning by Learning Differentiable Physics Models from Video and Languagecited by 55
  19. 2021Neuro-Symbolic Artificial Intelligence: Current Trendscited by 54
  20. 2023See, Think, Confirm: Interactive Prompting Between Vision and Language Models for Knowledge-based Visual Reasoningcited by 51
  21. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 48
  22. 2021Leveraging Language to Learn Program Abstractions and Search Heuristicscited by 45
  23. 2021Abstract Spatial-Temporal Reasoning via Probabilistic Abduction and Executioncited by 43
  24. 2023Encoding Time-Series Explanations through Self-Supervised Model Behavior Consistencycited by 42
  25. 2023Neither hype nor gloom do DNNs justicecited by 40
  26. 2021A Survey on Neural-symbolic Learning Systemscited by 39
  27. 2021Systematic Generalization with Edge Transformerscited by 38
  28. 2022Is Neuro-Symbolic AI Meeting its Promise in Natural Language Processing? A Structured Reviewcited by 38
  29. 2025Spatial457: A Diagnostic Benchmark for 6D Spatial Reasoning of Large Multimodal Modelscited by 38
  30. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  31. 2021Efficient Iterative Amortized Inference for Learning Symmetric and Disentangled Multi-Object Representationscited by 36
  32. 2022MultiViz: Towards Visualizing and Understanding Multimodal Modelscited by 36
  33. 2021DualVGR: A Dual-Visual Graph Reasoning Unit for Video Question Answeringcited by 35
  34. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 35
  35. 2024Can Large Language Models Understand Symbolic Graphics Programs?cited by 35
  36. 2021Greedy Gradient Ensemble for Robust Visual Question Answeringcited by 34
  37. 2021PTR: A Benchmark for Part-based Conceptual, Relational, and Physical Reasoningcited by 34
  38. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34

50 most cited of 372 All 372 (JSON)