# lamBERT: Language and Action Learning Using Multimodal BERT

arXiv: https://arxiv.org/abs/2004.07093 · HTML: https://mlsanity.com/paper/2004.07093/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Apr 2020.

## Built on by (most cited first, 3 of 3)

- [Are We There Yet? Learning to Localize in Embodied Instruction Following](https://mlsanity.com/paper/2101.03431/) (2021)
- [A Whole Brain Probabilistic Generative Model: Toward Realizing Cognitive Architectures for Developmental Robots](https://mlsanity.com/paper/2103.08183/) (2021)
- [DiMBERT: Learning Vision-Language Grounded Representations with Disentangled Multimodal-Attention](https://mlsanity.com/paper/2210.16431/) (2022)

## Abstract (arXiv)

Recently, the bidirectional encoder representations from transformers (BERT) model has attracted much attention in the field of natural language processing, owing to its high performance in language understanding-related tasks. The BERT model learns language representation that can be adapted to various tasks via pre-training using a large corpus in an unsupervised manner. This study proposes the language and action learning using multimodal BERT (lamBERT) model that enables the learning of language and actions by 1) extending the BERT model to multimodal representation and 2) integrating it with reinforcement learning. To verify the proposed model, an experiment is conducted in a grid environment that requires language understanding for the agent to act properly. As a result, the lamBERT model obtained higher rewards in multitask settings and transfer settings when compared to other models, such as the convolutional neural network-based model and the lamBERT model without pre-training.