mlsanityData to 7 Oct 2026

BERTGEN: Multi-task Generation through BERT

arXiv 2106.03484 ↗First posted 7 Jun 2021cs.CLcited by 3

Not on the map. Why?

Abstract (arXiv)

We present BERTGEN, a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pretrained models VL-BERT and M-BERT, respectively. BERTGEN is auto-regressively trained for language generation tasks, namely image captioning, machine translation and multimodal machine translation, under a multitask setting. With a comprehensive set of evaluations, we show that BERTGEN outperforms many strong baselines across the tasks explored. We also show BERTGEN's ability for zero-shot language generation, where it exhibits competitive performance to supervised counterparts. Finally, we conduct ablation studies which demonstrate that BERTGEN substantially benefits from multi-tasking and effectively transfers relevant inductive biases from the pre-trained models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 36 of its 56 references that are in the map. First posted on arXiv 7 Jun 2021.

Builds on

36 of 56 references.

References · 36

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2014Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translationcited by 4,016
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2018Deep contextualized word representationscited by 2,285
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  10. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
24 more
  1. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  2. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  3. 2019VideoBERT: A Joint Model for Video and Language Representation Learningcited by 702
  4. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  5. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  6. 2016Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translationcited by 467
  7. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  8. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  9. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  10. 2016Multi30K: Multilingual English-German Image Descriptionscited by 250
  11. 2016Knowing When to Look: Adaptive Attention via A Visual Sentinel for Image Captioningcited by 231
  12. 2019Leveraging Pre-trained Checkpoints for Sequence Generation Taskscited by 182
  13. 2016Multi-Way, Multilingual Neural Machine Translation with a Shared Attention Mechanismcited by 117
  14. 2018Neural Baby Talkcited by 107
  15. 2017Learned in Translation: Contextualized Word Vectorscited by 104
  16. 2015Multi-task Sequence to Sequence Learningcited by 94
  17. 2017One Model To Learn Them Allcited by 86
  18. 2016Toward Multilingual Neural Machine Translation with Universal Encoder and Decodercited by 76
  19. 2019Probing the Need for Visual Context in Multimodal Machine Translationcited by 54
  20. 2019Towards Making the Most of BERT in Neural Machine Translationcited by 43
  21. 2016Multi-Source Neural Translationcited by 37
  22. 2019Multimodal Machine Translation through Visuals and Speechcited by 29
  23. 2019On the use of BERT for Neural Machine Translationcited by 23
  24. 2020Simultaneous Machine Translation with Visual Contextcited by 17

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 1; 2024: 0; 2025: 1.

3 papers.

  1. 2023Towards a Unified Model for Generating Answers and Explanations in Visual Question Answeringcited by 3
  2. 2022Multilingual Multimodality: A Taxonomical Survey of Datasets, Techniques, Challenges and Opportunitiescited by 1
  3. 2025Scaling Laws for Conditional Emergence of Multilingual Image Captioning via Generalization from Translationcited by 0