mlsanityData to 14 Oct 2026

Multimodal Sequential Generative Models for Semi-Supervised Language Instruction Following

arXiv 2301.00676 ↗First posted 29 Dec 2022cs.LG cs.AI cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Agents that can follow language instructions are expected to be useful in a variety of situations such as navigation. However, training neural network-based agents requires numerous paired trajectories and languages. This paper proposes using multimodal generative models for semi-supervised learning in the instruction following tasks. The models learn a shared representation of the paired data, and enable semi-supervised learning by reconstructing unpaired data through the representation. Key challenges in applying the models to sequence-to-sequence tasks including instruction following are learning a shared representation of variable-length mulitimodal data and incorporating attention mechanisms. To address the problems, this paper proposes a novel network architecture to absorb the difference in the sequence lengths of the multimodal data. In addition, to further improve the performance, this paper shows how to incorporate the generative model-based approach with an existing semi-supervised method called a speaker-follower model, and proposes a regularization term that improves inference using unpaired trajectories. Experiments on BabyAI and Room-to-Room (R2R) environments show that the proposed method improves the performance of instruction following by leveraging unpaired data, and improves the performance of the speaker-follower model by 2% to 4% in R2R.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 27 references that are in the map. First posted on arXiv 29 Dec 2022.

Builds on

20 of 27 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  3. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,264
  4. 2017Vision-and-Language Navigation: Interpreting visually-grounded navigation instructions in real environmentscited by 1,027
  5. 2015Effective Approaches to Attention-based Neural Machine Translationcited by 1,026
  6. 2018Speaker-Follower Models for Vision-and-Language Navigationcited by 340
  7. 2018Reinforced Cross-Modal Matching and Self-Supervised Imitation Learning for Vision-Language Navigationcited by 305
  8. 2020A Recurrent Vision-and-Language BERT for Navigationcited by 302
  9. 2017Unsupervised Machine Translation Using Monolingual Corpora Onlycited by 273
  10. 2019Learning to Navigate Unseen Environments: Back Translation with Environmental Dropoutcited by 244
  11. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  12. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 178
8 more
  1. 2017Unsupervised Neural Machine Translationcited by 149
  2. 2018Set Transformer: A Framework for Attention-based Permutation-Invariant Neural Networkscited by 132
  3. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  4. 2018Doubly Reparameterized Gradient Estimators for Monte Carlo Objectivescited by 41
  5. 2019Counterfactual Vision-and-Language Navigation via Adversarial Path Samplingcited by 22
  6. 2019Multi-modal Discriminative Model for Vision-and-Language Navigationcited by 17
  7. 2016Semantic Parsing with Semi-Supervised Sequential Autoencoderscited by 11
  8. 2020Take the Scenic Route: Improving Generalization in Vision-and-Language Navigationcited by 10

Built on by

Not cited yet.