mlsanityData to 14 Oct 2026

LXMERT: Learning Cross-Modality Encoder Representations from Transformers

arXiv 1908.07490 ↗First posted 20 Aug 2019cs.CL cs.CV cs.LGcited by 1,553

Not on the map. Why?

Cited by 290 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Vision-and-language reasoning requires an understanding of visual concepts, language semantics, and, most importantly, the alignment and relationships between these two modalities. We thus propose the LXMERT (Learning Cross-Modality Encoder Representations from Transformers) framework to learn these vision-and-language connections. In LXMERT, we build a large-scale Transformer model that consists of three encoders: an object relationship encoder, a language encoder, and a cross-modality encoder. Next, to endow our model with the capability of connecting vision and language semantics, we pre-train the model with large amounts of image-and-sentence pairs, via five diverse representative pre-training tasks: masked language modeling, masked object prediction (feature regression and label classification), cross-modality matching, and image question answering. These tasks help in learning both intra-modality and cross-modality relationships. After fine-tuning from our pre-trained parameters, our model achieves the state-of-the-art results on two visual question answering datasets (i.e., VQA and GQA). We also show the generalizability of our pre-trained cross-modality model by adapting it to a challenging visual-reasoning task, NLVR2, and improve the previous best result by 22% absolute (54% to 76%). Lastly, we demonstrate detailed ablation studies to prove that both our novel model components and pre-training strategies significantly contribute to our strong results; and also present several attention visualizations for the different encoders. Code and pre-trained models publicly available at: https://github.com/airsplay/lxmert

Key facts. Cited by 1,553 papers in the map (data through 14 Oct 2026). First posted on arXiv 20 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 304; 2022: 387; 2023: 367; 2024: 249; 2025: 166; 2026: 80.

1,553 papers.

Sort:

By area:

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  4. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,080
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,619
  7. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  8. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  9. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  10. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  11. 2021Grounded Language-Image Pre-trainingcited by 1,156
  12. 2021CLIP-Adapter: Better Vision-Language Models with Feature Adapterscited by 1,026
38 more
  1. 2021Transformers in Vision: A Surveycited by 1,013
  2. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
  3. 2023LRM: Large Reconstruction Model for Single Image to 3Dcited by 849
  4. 2021MDETR -- Modulated Detection for End-to-End Multi-Modal Understandingcited by 819
  5. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 766
  6. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  7. 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
  8. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  9. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  10. 2022MTEB: Massive Text Embedding Benchmarkcited by 640
  11. 2021CLIPort: What and Where Pathways for Robotic Manipulationcited by 622
  12. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 616
  13. 2023ViperGPT: Visual Inference via Python Execution for Reasoningcited by 603
  14. 2021ClipCap: CLIP Prefix for Image Captioningcited by 559
  15. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  16. 2022Winoground: Probing Vision and Language Models for Visio-Linguistic Compositionalitycited by 529
  17. 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 512
  18. 2022LayoutLMv3: Pre-training for Document AI with Unified Text and Image Maskingcited by 485
  19. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  20. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 431
  21. 2023TIFA: Accurate and Interpretable Text-to-Image Faithfulness Evaluation with Question Answeringcited by 428
  22. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  23. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 421
  24. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  25. 2022Multimodal Learning with Transformers: A Surveycited by 400
  26. 2022Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classificationcited by 362
  27. 2022Prompt-aligned Gradient for Prompt Tuningcited by 341
  28. 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Modelscited by 339
  29. 2021An End-to-End Transformer Model for 3D Object Detectioncited by 329
  30. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  31. 2021Generic Attention-model Explainability for Interpreting Bi-Modal and Encoder-Decoder Transformerscited by 320
  32. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  33. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  34. 2022Language in a Bottle: Language Model Guided Concept Bottlenecks for Interpretable Image Classificationcited by 312
  35. 2021MERLOT: Multimodal Neural Script Knowledge Modelscited by 311
  36. 2022Generalized Decoding for Pixel, Image, and Languagecited by 294
  37. 2022Visual Spatial Reasoningcited by 293
  38. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 291

50 most cited of 1,553 All 1,553 (JSON)