# What is Learned in Visually Grounded Neural Syntax Acquisition

arXiv: https://arxiv.org/abs/2005.01678 · HTML: https://mlsanity.com/paper/2005.01678/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 4 May 2020.

## Built on by (most cited first, 15 of 15)

- [Vision-Language Intelligence: Tasks, Representation Learning, and Large Models](https://mlsanity.com/paper/2203.01922/) (2022)
- [Is BERT Blind? Exploring the Effect of Vision-and-Language Pretraining on Visual Language Understanding](https://mlsanity.com/paper/2303.12513/) (2023)
- [Mitigating Open-Vocabulary Caption Hallucinations](https://mlsanity.com/paper/2312.03631/) (2023)
- [Towers of Babel: Combining Images, Language, and 3D Geometry for Learning Multimodal Vision](https://mlsanity.com/paper/2108.05863/) (2021)
- [Video-aided Unsupervised Grammar Induction](https://mlsanity.com/paper/2104.04369/) (2021)
- [Kiki or Bouba? Sound Symbolism in Vision-and-Language Models](https://mlsanity.com/paper/2310.16781/) (2023)
- [KANDINSKYPatterns -- An experimental exploration environment for Pattern Analysis and Machine Intelligence](https://mlsanity.com/paper/2103.00519/) (2021)
- [A Surprising Failure? Multimodal LLMs and the NLVR Challenge](https://mlsanity.com/paper/2402.17793/) (2024)
- [Dependency Induction Through the Lens of Visual Perception](https://mlsanity.com/paper/2109.09790/) (2021)
- [Re-evaluating the Need for Multimodal Signals in Unsupervised Grammar Induction](https://mlsanity.com/paper/2212.10564/) (2022)
- [A Joint Study of Phrase Grounding and Task Performance in Vision and Language Models](https://mlsanity.com/paper/2309.02691/) (2023)
- [Learning Language Structures through Grounding](https://mlsanity.com/paper/2406.09662/) (2024)
- [Does Vision Accelerate Hierarchical Generalization in Neural Language Learners?](https://mlsanity.com/paper/2302.00667/) (2023)
- [Neural Abstructions: Abstractions that Support Construction for Grounded Language Learning](https://mlsanity.com/paper/2107.09285/) (2021)
- [ICC: Quantifying Image Caption Concreteness for Multimodal Dataset Curation](https://mlsanity.com/paper/2403.01306/) (2024)

## Abstract (arXiv)

Visual features are a promising signal for learning bootstrap textual models. However, blackbox learning models make it difficult to isolate the specific contribution of visual components. In this analysis, we consider the case study of the Visually Grounded Neural Syntax Learner (Shi et al., 2019), a recent approach for learning syntax from a visual training signal. By constructing simplified versions of the model, we isolate the core factors that yield the model's strong performance. Contrary to what the model might be capable of learning, we find significantly less expressive versions produce similar predictions and perform just as well, or even better. We also find that a simple lexical signal of noun concreteness plays the main role in the model's predictions as opposed to more complex syntactic reasoning.