# Neural Variational Learning for Grounded Language Acquisition

arXiv: https://arxiv.org/abs/2107.14593 · HTML: https://mlsanity.com/paper/2107.14593/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 63 references that are in the map.
- First posted on arXiv 20 Jul 2021.

## Builds on (most cited first, 30 of 30)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [Learning Transferable Architectures for Scalable Image Recognition](https://mlsanity.com/paper/1707.07012/) (2017)
- [Semi-Supervised Learning with Deep Generative Models](https://mlsanity.com/paper/1406.5298/) (2014)
- [Learning Hand-Eye Coordination for Robotic Grasping with Deep Learning and Large-Scale Data Collection](https://mlsanity.com/paper/1603.02199/) (2016)
- [Experience Grounds Language](https://mlsanity.com/paper/2004.10151/) (2020)
- [SEED: Semantics Enhanced Encoder-Decoder Framework for Scene Text Recognition](https://mlsanity.com/paper/2005.10977/) (2020)
- [Grounding of Textual Phrases in Images by Reconstruction](https://mlsanity.com/paper/1511.03745/) (2015)
- [Text2Action: Generative Adversarial Synthesis from Language to Action](https://mlsanity.com/paper/1710.05298/) (2017)
- [MART: Memory-Augmented Recurrent Transformer for Coherent Video Paragraph Captioning](https://mlsanity.com/paper/2005.05402/) (2020)
- [Variational Autoencoders for Deforming 3D Mesh Models](https://mlsanity.com/paper/1709.04307/) (2017)
- [Interactively Picking Real-World Objects with Unconstrained Spoken Language Instructions](https://mlsanity.com/paper/1710.06280/) (2017)
- [Learning a bidirectional mapping between human whole-body motion and natural language using deep recurrent neural networks](https://mlsanity.com/paper/1705.06400/) (2017)
- [Reusing Discriminators for Encoding: Towards Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2003.00273/) (2020)
- [Just Ask:An Interactive Learning Framework for Vision and Language Navigation](https://mlsanity.com/paper/1912.00915/) (2019)
- [Object Referring in Videos with Language and Human Gaze](https://mlsanity.com/paper/1801.01582/) (2018)
- [Learning the Redundancy-free Features for Generalized Zero-Shot Object Recognition](https://mlsanity.com/paper/2006.08939/) (2020)
- [Words aren't enough, their order matters: On the Robustness of Grounding Visual Referring Expressions](https://mlsanity.com/paper/2005.01655/) (2020)
- [Symbol Emergence in Robotics: A Survey](https://mlsanity.com/paper/1509.08973/) (2015)
- [Robot Object Retrieval with Contextual Natural Language Queries](https://mlsanity.com/paper/2006.13253/) (2020)
- [Task Aligned Generative Meta-learning for Zero-shot Learning](https://mlsanity.com/paper/2103.02185/) (2021)
- [Crossing Nets: Combining GANs and VAEs with a Shared Latent Space for Hand Pose Estimation](https://mlsanity.com/paper/1702.03431/) (2017)
- [Hierarchical Photo-Scene Encoder for Album Storytelling](https://mlsanity.com/paper/1902.00669/) (2019)
- [Mix and match networks: encoder-decoder alignment for zero-pair image translation](https://mlsanity.com/paper/1804.02199/) (2018)

## Built on by (most cited first, 1 of 1)

- [Learning the meanings of function words from grounded language using a visual question answering model](https://mlsanity.com/paper/2308.08628/) (2023)

## Abstract (arXiv)

We propose a learning system in which language is grounded in visual percepts without specific pre-defined categories of terms. We present a unified generative method to acquire a shared semantic/visual embedding that enables the learning of language about a wide range of real-world objects. We evaluate the efficacy of this learning by predicting the semantics of objects and comparing the performance with neural and non-neural inputs. We show that this generative approach exhibits promising results in language grounding without pre-specifying visual categories under low resource settings. Our experiments demonstrate that this approach is generalizable to multilingual, highly varied datasets.