# VSA4VQA: Scaling a Vector Symbolic Architecture to Visual Question Answering on Natural Images

arXiv: https://arxiv.org/abs/2405.03852 · HTML: https://mlsanity.com/paper/2405.03852/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 10 of its 18 references that are in the map.
- First posted on arXiv 6 May 2024.

## Builds on (most cited first, 10 of 10)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [What does CLIP know about a red circle? Visual prompt engineering for VLMs](https://mlsanity.com/paper/2304.06712/) (2023)
- [ReCLIP: A Strong Zero-Shot Baseline for Referring Expression Comprehension](https://mlsanity.com/paper/2204.05991/) (2022)
- [CLIP Models are Few-shot Learners: Empirical Studies on VQA and Visual Entailment](https://mlsanity.com/paper/2203.07190/) (2022)
- [A Neuro-vector-symbolic Architecture for Solving Raven's Progressive Matrices](https://mlsanity.com/paper/2203.04571/) (2022)
- [Meta Module Network for Compositional Visual Reasoning](https://mlsanity.com/paper/1910.03230/) (2019)
- [Weakly Supervised Relative Spatial Reasoning for Visual Question Answering](https://mlsanity.com/paper/2109.01934/) (2021)
- [Modularized Zero-shot VQA with Pre-trained Models](https://mlsanity.com/paper/2305.17369/) (2023)

## Abstract (arXiv)

While Vector Symbolic Architectures (VSAs) are promising for modelling spatial cognition, their application is currently limited to artificially generated images and simple spatial queries. We propose VSA4VQA - a novel 4D implementation of VSAs that implements a mental representation of natural images for the challenging task of Visual Question Answering (VQA). VSA4VQA is the first model to scale a VSA to complex spatial queries. Our method is based on the Semantic Pointer Architecture (SPA) to encode objects in a hyperdimensional vector space. To encode natural images, we extend the SPA to include dimensions for object's width and height in addition to their spatial location. To perform spatial queries we further introduce learned spatial query masks and integrate a pre-trained vision-language model for answering attribute-related questions. We evaluate our method on the GQA benchmark dataset and show that it can effectively encode natural images, achieving competitive performance to state-of-the-art deep learning methods for zero-shot VQA.