mlsanityData to 14 Oct 2026

Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformers

arXiv 2004.00849 ↗First posted 2 Apr 2020cs.CV cs.CL cs.LGcited by 288

Not on the map. Why?

Cited by 69 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We propose Pixel-BERT to align image pixels with text by deep multi-modal transformers that jointly learn visual and language embedding in a unified end-to-end framework. We aim to build a more accurate and thorough connection between image pixels and language semantics directly from image and sentence pairs instead of using region-based image features as the most recent vision and language tasks. Our Pixel-BERT which aligns semantic connection in pixel and text level solves the limitation of task-specific visual representation for vision and language tasks. It also relieves the cost of bounding box annotations and overcomes the unbalance between semantic labels in visual task and language semantic. To provide a better representation for down-stream tasks, we pre-train a universal end-to-end model with image and sentence pairs from Visual Genome dataset and MS-COCO dataset. We propose to use a random pixel sampling mechanism to enhance the robustness of visual representation and to apply the Masked Language Model and Image-Text Matching as pre-training tasks. Extensive experiments on downstream tasks with our pre-trained model show that our approach makes the most state-of-the-arts in downstream tasks, including Visual Question Answering (VQA), image-text retrieval, Natural Language for Visual Reasoning for Real (NLVR). Particularly, we boost the performance of a single model in VQA task by 2.17 points compared with SOTA under fair comparison.

Key facts. Cited by 288 papers in the map (data through 14 Oct 2026). First posted on arXiv 2 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 71; 2022: 88; 2023: 77; 2024: 35; 2025: 15; 2026: 2.

288 papers.

Sort:

By area:

  1. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  2. 2021ViLT: Vision-and-Language Transformer Without Convolution or Region Supervisioncited by 1,230
  3. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  5. 2023The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)cited by 681
  6. 2021FLAVA: A Foundational Language And Vision Alignment Modelcited by 649
  7. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 570
  8. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537
  9. 2021Less is More: ClipBERT for Video-and-Language Learning via Sparse Samplingcited by 484
  10. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 425
  11. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 412
  12. 2021Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-training Paradigmcited by 405
38 more
  1. 2022Multimodal Learning with Transformers: A Surveycited by 400
  2. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  3. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 320
  4. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 305
  5. 2024An Image is Worth 32 Tokens for Reconstruction and Generationcited by 302
  6. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
  7. 2021Multi-Grained Vision Language Pre-Training: Aligning Texts with Visual Conceptscited by 256
  8. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  9. 2022Zero-Shot Video Question Answering via Frozen Bidirectional Language Modelscited by 229
  10. 2021Advancing High-Resolution Video-Language Representation with Large-Scale Video Transcriptionscited by 226
  11. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 224
  12. 2021Scaling Up Vision-Language Pre-training for Image Captioningcited by 196
  13. 2021Seeing Out of tHe bOx: End-to-End Pre-training for Vision-Language Representation Learningcited by 187
  14. 2022mPLUG: Effective and Efficient Vision-Language Learning by Cross-modal Skip-connectionscited by 187
  15. 2021Combined Scaling for Zero-shot Transfer Learningcited by 182
  16. 2021RedCaps: web-curated image-text data created by the people, for the peoplecited by 180
  17. 2022A Survey of Vision-Language Pre-Trained Modelscited by 177
  18. 2023Towards Open Vocabulary Learning: A Surveycited by 166
  19. 2022SegCLIP: Patch Aggregation with Learnable Centers for Open-Vocabulary Semantic Segmentationcited by 164
  20. 2023UniChart: A Universal Vision-language Pretrained Model for Chart Comprehension and Reasoningcited by 160
  21. 2022VLP: A Survey on Vision-Language Pre-trainingcited by 159
  22. 2025Sa2VA: Marrying SAM2 with MLLM for Dense Grounded Understanding of Images and Videoscited by 159
  23. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  24. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
  25. 2022Unifying Vision, Text, and Layout for Universal Document Processingcited by 140
  26. 2021Recent Advances in Deep Learning Based Dialogue Systems: A Systematic Surveycited by 130
  27. 2021Uni-Perceiver: Pre-training Unified Architecture for Generic Perception for Zero-shot and Few-shot Taskscited by 125
  28. 2021UniTAB: Unifying Text and Box Outputs for Grounded Vision-Language Modelingcited by 122
  29. 2022Vision-Language Pre-training: Basics, Recent Advances, and Future Trendscited by 117
  30. 2022VLT: Vision-Language Transformer and Query Generation for Referring Segmentationcited by 111
  31. 2021VinVL: Revisiting Visual Representations in Vision-Language Modelscited by 110
  32. 2021Probing Image-Language Transformers for Verb Understandingcited by 109
  33. 2022Coarse-to-Fine Vision-Language Pre-training with Fusion in the Backbonecited by 109
  34. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 109
  35. 2022Multi-Modal Masked Autoencoders for Medical Vision-and-Language Pre-Trainingcited by 104
  36. 2023DRESS: Instructing Large Vision-Language Models to Align and Interact with Humans via Natural Language Feedbackcited by 103
  37. 2022TubeDETR: Spatio-Temporal Video Grounding with Transformerscited by 98
  38. 2021Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Trainingcited by 96

50 most cited of 288 All 288 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2023Language Quantized AutoEncoders: Towards Unsupervised Text-Image Alignment
  2. 2024From Pixels to Tokens: Byte-Pair Encoding on Quantized Visual Modalities
  3. 2025PixCLIP: Achieving Fine-grained Visual Language Understanding via Any-granularity Pixel-Text Alignment Learning
  4. 2023TagAlign: Improving Vision-Language Alignment with Multi-Tag Classification
  5. 2026SPAR: Semantic-Pixel Self-Alignment and Adaptive Routing for Unified Multimodal Models
  6. 2025BASIC: Boosting Visual Alignment with Intrinsic Refined Embeddings in Multimodal Large Language Models
  7. 2025Growing Visual Generative Capacity for Pre-Trained MLLMs
  8. 2026On the Design Fundamentals of Pixel Text Representation Learning