mlsanityData to 14 Oct 2026

PaLI: A Jointly-Scaled Multilingual Language-Image Model

arXiv 2209.06794 ↗First posted 14 Sep 2022cs.CV cs.CLcited by 766

Not on the map. Why?

Cited by 287 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Effective scaling and a flexible task interface enable large language models to excel at many tasks. We present PaLI (Pathways Language and Image model), a model that extends this approach to the joint modeling of language and vision. PaLI generates text based on visual and textual inputs, and with this interface performs many vision, language, and multimodal tasks, in many languages. To train PaLI, we make use of large pre-trained encoder-decoder language models and Vision Transformers (ViTs). This allows us to capitalize on their existing capabilities and leverage the substantial cost of training them. We find that joint scaling of the vision and language components is important. Since existing Transformers for language are much larger than their vision counterparts, we train a large, 4-billion parameter ViT (ViT-e) to quantify the benefits from even larger-capacity vision models. To train PaLI, we create a large multilingual mix of pretraining tasks, based on a new image-text training set containing 10B images and texts in over 100 languages. PaLI achieves state-of-the-art in multiple vision and language tasks (such as captioning, visual question-answering, scene-text understanding), while retaining a simple, modular, and scalable design.

Key facts. Cited by 766 papers in the map (data through 14 Oct 2026). Builds on 82 of its 112 references that are in the map. First posted on arXiv 14 Sep 2022.

Builds on

82 of 112 references.

References · 82

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  6. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  7. 2020Scaling Laws for Neural Language Modelscited by 6,586
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  11. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  12. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,656
38 more
  1. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,516
  2. 2015VQA: Visual Question Answeringcited by 2,506
  3. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,046
  4. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  5. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,850
  6. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,818
  7. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  8. 2019Towards VQA Models That Can Readcited by 1,718
  9. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,553
  10. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,515
  11. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,462
  12. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  13. 2019Natural Adversarial Examplescited by 1,408
  14. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  15. 2018Datasheets for Datasetscited by 1,386
  16. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,274
  17. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  18. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,121
  19. 2018Model Cards for Model Reportingcited by 1,080
  20. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,013
  21. 2021Scaling Vision Transformerscited by 999
  22. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 968
  23. 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 915
  24. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 905
  25. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  26. 2014Show and Tell: A Neural Image Caption Generatorcited by 879
  27. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 858
  28. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  29. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  30. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  31. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  32. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 742
  33. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 647
  34. 2018Exploring the Limits of Weakly Supervised Pretrainingcited by 605
  35. 2019On the Cross-lingual Transferability of Monolingual Representationscited by 578
  36. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 577
  37. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  38. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 537

50 most cited of 82 All 82 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 24; 2023: 240; 2024: 247; 2025: 182; 2026: 73.

766 papers.

Sort:

By area:

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,137
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  5. 2024OpenVLA: An Open-Source Vision-Language-Action Modelcited by 2,887
  6. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  7. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,224
  8. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  9. 2022Objaverse: A Universe of Annotated 3D Objectscited by 1,540
  10. 2022Reproducible scaling laws for contrastive language-image learningcited by 1,173
  11. 2023Vision-Language Models for Vision Tasks: A Surveycited by 839
  12. 2024Cambrian-1: A Fully Open, Vision-Centric Exploration of Multimodal LLMscited by 838
38 more
  1. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 712
  2. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 708
  3. 2024SpatialVLM: Endowing Vision-Language Models with Spatial Reasoning Capabilitiescited by 706
  4. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 694
  5. 2023Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decodingcited by 688
  6. 2023DataComp: In search of the next generation of multimodal datasetscited by 644
  7. 2023Language Model Beats Diffusion -- Tokenizer is Key to Visual Generationcited by 640
  8. 2023OpenFlamingo: An Open-Source Framework for Training Large Autoregressive Vision-Language Modelscited by 565
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  10. 2023mPLUG-Owl2: Revolutionizing Multi-modal Large Language Model with Modality Collaborationcited by 547
  11. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  12. 2025SmolVLA: A Vision-Language-Action Model for Affordable and Efficient Roboticscited by 530
  13. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 509
  14. 2024Evaluating Text-to-Visual Generation with Image-to-Text Generationcited by 474
  15. 2023Learning Interactive Real-World Simulatorscited by 458
  16. 2023Directly Fine-Tuning Diffusion Models on Differentiable Rewardscited by 448
  17. 2023RLHF-V: Towards Trustworthy MLLMs via Behavior Alignment from Fine-grained Correctional Human Feedbackcited by 412
  18. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  19. 2023EVA-02: A Visual Representation for Neon Genesiscited by 381
  20. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 366
  21. 2023Scaling Open-Vocabulary Object Detectioncited by 343
  22. 2023AudioPaLM: A Large Language Model That Can Speak and Listencited by 343
  23. 2025Perception Encoder: The best visual embeddings are not at the output of the networkcited by 326
  24. 2023DreamLLM: Synergistic Multimodal Comprehension and Creationcited by 297
  25. 2023Towards Generalist Biomedical AIcited by 289
  26. 2024Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoningcited by 280
  27. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 279
  28. 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 278
  29. 2023Vid2Seq: Large-Scale Pretraining of a Visual Language Model for Dense Video Captioningcited by 263
  30. 2023PaLI-X: On Scaling up a Multilingual Vision and Language Modelcited by 255
  31. 2023From Sparse to Soft Mixtures of Expertscited by 248
  32. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  33. 2023SugarCrepe: Fixing Hackable Benchmarks for Vision-Language Compositionalitycited by 243
  34. 2024Capabilities of Gemini Models in Medicinecited by 237
  35. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 231
  36. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  37. 2024Sapiens: Foundation for Human Vision Modelscited by 229
  38. 2023LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Modelscited by 217

50 most cited of 766 All 766 (JSON)