mlsanityData to 14 Oct 2026

Learning by Abstraction: The Neural State Machine

arXiv 1907.03950 ↗First posted 9 Jul 2019cs.AI cs.CL cs.CVcited by 113

Not on the map. Why?

Abstract (arXiv)

We introduce the Neural State Machine, seeking to bridge the gap between the neural and symbolic views of AI and integrate their complementary strengths for the task of visual reasoning. Given an image, we first predict a probabilistic graph that represents its underlying semantics and serves as a structured world model. Then, we perform sequential reasoning over the graph, iteratively traversing its nodes to answer a given question or draw a new inference. In contrast to most neural architectures that are designed to closely interact with the raw sensory data, our model operates instead in an abstract latent space, by transforming both the visual and linguistic modalities into semantic concept-based representations, thereby achieving enhanced transparency and modularity. We evaluate our model on VQA-CP and GQA, two recent VQA datasets that involve compositionality, multi-step inference and diverse reasoning skills, achieving state-of-the-art results in both cases. We provide further experiments that illustrate the model's strong generalization capacity across multiple dimensions, including novel compositions of concepts, changes in the answer distribution, and unseen linguistic structures, demonstrating the qualities and efficacy of our approach.

Key facts. Cited by 113 papers in the map (data through 14 Oct 2026). First posted on arXiv 9 Jul 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 41; 2022: 30; 2023: 25; 2024: 10; 2025: 6; 2026: 1.

113 papers.

Sort:

By area:

  1. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  2. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  3. 2022Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoningcited by 334
  4. 2021CPT: Colorful Prompt Tuning for Pre-trained Vision-Language Modelscited by 183
  5. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  6. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  7. 2022UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graphcited by 103
  8. 2021Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signalscited by 101
  9. 2022Exploiting Unlabeled Data with Vision and Language Models for Object Detectioncited by 97
  10. 2021A Survey on Complex Knowledge Base Question Answering: Methods, Challenges and Solutionscited by 78
  11. 2021Grounding Physical Concepts of Objects and Events Through Dynamic Visual Reasoningcited by 68
  12. 2021Learning to Generate Scene Graph from Natural Language Supervisioncited by 64
38 more
  1. 2023NS3D: Neuro-Symbolic Grounding of 3D Objects and Relationscited by 57
  2. 2023Interpretable Neural-Symbolic Concept Reasoningcited by 57
  3. 2021Complex Knowledge Base Question Answering: A Surveycited by 52
  4. 2024From Pixels to Graphs: Open-Vocabulary Scene Graph Generation with Vision-Language Modelscited by 43
  5. 2021A Survey on Neural-symbolic Learning Systemscited by 39
  6. 2022Towards Data-and Knowledge-Driven Artificial Intelligence: A Survey on Neuro-Symbolic Computingcited by 35
  7. 2023What's Left? Concept Grounding with Logic-Enhanced Foundation Modelscited by 33
  8. 2023Dissociating language and thought in large language modelscited by 29
  9. 2022HAKE: A Knowledge Engine Foundation for Human Activity Understandingcited by 28
  10. 2021Learning Algebraic Representation for Systematic Generalization in Abstract Reasoningcited by 27
  11. 2021Visual Distant Supervision for Scene Graph Generationcited by 25
  12. 2021COVR: A test-bed for Visually Grounded Compositional Generalization with real imagescited by 25
  13. 2021Coarse-to-Fine Reasoning for Visual Question Answeringcited by 25
  14. 2021Separating Skills and Concepts for Novel Visual Question Answeringcited by 24
  15. 2021Compositional Transformers for Scene Generationcited by 24
  16. 2022Are Deep Neural Networks SMARTer than Second Graders?cited by 24
  17. 2023ReasoningLM: Enabling Structural Subgraph Reasoning in Pre-trained Language Models for Question Answering over Knowledge Graphcited by 23
  18. 2021ProTo: Program-Guided Transformer for Program-Guided Taskscited by 21
  19. 2023Visual Question Answering: A Survey on Techniques and Common Trends in Recent Literaturecited by 21
  20. 2021Understanding the Role of Scene Graphs in Visual Question Answeringcited by 20
  21. 2021SemVLP: Vision-Language Pre-training by Aligning Semantics at Multiple Levelscited by 19
  22. 2021COMPOSER: Compositional Reasoning of Group Activity in Videos with Keypoint-Only Modalitycited by 19
  23. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  24. 2021Comprehensive Visual Question Answering on Point Clouds through Compositional Scene Manipulationcited by 17
  25. 2023Grounded Object Centric Learningcited by 17
  26. 2021GraghVQA: Language-Guided Graph Neural Networks for Graph-based Visual Question Answeringcited by 16
  27. 2022CLEVR-X: A Visual Reasoning Dataset for Natural Language Explanationscited by 16
  28. 2022ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphscited by 16
  29. 2023Symbol-LLM: Leverage Language Models for Symbolic System in Visual Human Activity Reasoningcited by 16
  30. 2021Are Bias Mitigation Techniques for Deep Learning Effective?cited by 14
  31. 2021Calibrating Concepts and Operations: Towards Symbolic Reasoning on Real Imagescited by 14
  32. 2022VQA and Visual Reasoning: An Overview of Recent Datasets, Methods and Challengescited by 14
  33. 2023Scalable Neural-Probabilistic Answer Set Programmingcited by 14
  34. 2021Graphhopper: Multi-Hop Scene Graph Reasoning for Visual Question Answeringcited by 13
  35. 2021Induce, Edit, Retrieve: Language Grounded Multimodal Schema for Instructional Video Retrievalcited by 13
  36. 2022Joint learning of object graph and relation graph for visual question answeringcited by 13
  37. 2021Hopper: Multi-hop Transformer for Spatiotemporal Reasoningcited by 12
  38. 2021Weakly Supervised Relative Spatial Reasoning for Visual Question Answeringcited by 12

50 most cited of 113 All 113 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Neural networks for abstraction and reasoning: Towards broad generalization in machines
  2. 2026Compositional Neuro-Symbolic Reasoning
  3. 2025Visual Graph Arena: Evaluating Visual Conceptualization of Vision and Multimodal Large Language Models
  4. 2025NePTune: A Neuro-Pythonic Framework for Tunable Compositional Reasoning on Vision-Language
  5. 2022Deep Neural Networks for Visual Reasoning
  6. 2022Visual Programming: Compositional visual reasoning without training
  7. 2023Abstract Visual Reasoning Enabled by Language
  8. 2026Symbolic Grounding Reveals Representational Bottlenecks in Abstract Visual Reasoning