mlsanityData to 7 Oct 2026

PaLI-X: On Scaling up a Multilingual Vision and Language Model

arXiv 2305.18565 ↗First posted 29 May 2023cs.CV cs.CL cs.LGcited by 254

Not on the map. Why?

Cited by 103 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We present the training recipe and results of scaling up PaLI-X, a multilingual vision and language model, both in terms of size of the components and the breadth of its training task mixture. Our model achieves new levels of performance on a wide-range of varied and complex tasks, including multiple image-based captioning and question-answering tasks, image-based document understanding and few-shot (in-context) learning, as well as object detection, video question answering, and video captioning. PaLI-X advances the state-of-the-art on most vision-and-language benchmarks considered (25+ of them). Finally, we observe emerging capabilities, such as complex counting and multilingual object detection, tasks that are not explicitly in the training mix.

Key facts. Cited by 254 papers in the map (data through 7 Oct 2026). Builds on 79 of its 95 references that are in the map. First posted on arXiv 29 May 2023.

Builds on

79 of 95 references.

References · 79

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  5. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  6. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2019Towards VQA Models That Can Readcited by 1,696
  11. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  12. 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
38 more
  1. 2019Natural Adversarial Examplescited by 1,401
  2. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  3. 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
  4. 2019LVIS: A Dataset for Large Vocabulary Instance Segmentationcited by 1,215
  5. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  6. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  7. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  8. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  9. 2021Open-vocabulary Object Detection via Vision and Language Knowledge Distillationcited by 903
  10. 2016Semantics derived automatically from language corpora contain human-like biasescited by 894
  11. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 890
  12. 2018VizWiz Grand Challenge: Answering Visual Questions from Blind Peoplecited by 852
  13. 2016A Diagram Is Worth A Dozen Imagescited by 793
  14. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  15. 2021NExT-QA:Next Phase of Question-Answering to Explaining Temporal Actionscited by 735
  16. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  17. 2021Vector-quantized Image Modeling with Improved VQGANcited by 656
  18. 2021SimVLM: Simple Visual Language Model Pretraining with Weak Supervisioncited by 645
  19. 2021RegionCLIP: Region-based Language-Image Pretrainingcited by 612
  20. 2019ActivityNet-QA: A Dataset for Understanding Complex Web Videos via Question Answeringcited by 559
  21. 2023Language Is Not All You Need: Aligning Perception with Language Modelscited by 543
  22. 2022GIT: A Generative Image-to-text Transformer for Vision and Languagecited by 536
  23. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  24. 2018nocaps: novel object captioning at scalecited by 444
  25. 2018Gender Bias in Coreference Resolutioncited by 438
  26. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  27. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  28. 2019VATEX: A Large-Scale, High-Quality Multilingual Dataset for Video-and-Language Researchcited by 412
  29. 2020TextCaps: a Dataset for Image Captioning with Reading Comprehensioncited by 398
  30. 2023Larger language models do in-context learning differentlycited by 386
  31. 2022Pix2Struct: Screenshot Parsing as Pretraining for Visual Language Understandingcited by 360
  32. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  33. 2020Are we done with ImageNet?cited by 329
  34. 2019Scene Text Visual Question Answeringcited by 327
  35. 2022Simple Open-Vocabulary Object Detection with Vision Transformerscited by 303
  36. 2022UL2: Unifying Language Learning Paradigmscited by 292
  37. 2019Towards Fairness in Visual Recognition: Effective Strategies for Bias Mitigationcited by 252
  38. 2018Women also Snowboard: Overcoming Bias in Captioning Modelscited by 236

50 most cited of 79 All 79 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 50; 2024: 95; 2025: 69; 2026: 40.

254 papers.

Sort:

By area:

  1. 2023Improved Baselines with Visual Instruction Tuningcited by 4,362
  2. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 2,993
  3. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  4. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  5. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  6. 2023CogVLM: Visual Expert for Pretrained Language Modelscited by 704
  7. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  8. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  9. 2023VILA: On Pre-training for Visual Language Modelscited by 686
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
  11. 2024BLINK: Multimodal Large Language Models Can See but Not Perceivecited by 529
  12. 2025SpatialVLA: Exploring Spatial Representations for Visual-Language-Action Modelcited by 502
38 more
  1. 2023Ferret: Refer and Ground Anything Anywhere at Any Granularitycited by 498
  2. 2023Learning Interactive Real-World Simulatorscited by 440
  3. 2024CogACT: A Foundational Vision-Language-Action Model for Synergizing Cognition and Action in Robotic Manipulationcited by 411
  4. 2023Florence-2: Advancing a Unified Representation for a Variety of Vision Taskscited by 403
  5. 2023RemoteCLIP: A Vision Language Foundation Model for Remote Sensingcited by 357
  6. 2023Monkey: Image Resolution and Text Label Are Important Things for Large Multi-modal Modelscited by 356
  7. 2023Vision-Language Foundation Models as Effective Robot Imitatorscited by 344
  8. 2024What matters when building vision-language models?cited by 321
  9. 2023Unified-IO 2: Scaling Autoregressive Multimodal Models with Vision, Language, Audio, and Actioncited by 301
  10. 2024VILA-U: a Unified Foundation Model Integrating Visual Understanding and Generationcited by 290
  11. 2024PyramidDrop: Accelerating Your Large Vision-Language Models via Pyramid Visual Redundancy Reductioncited by 283
  12. 2024A Survey on Vision-Language-Action Models for Embodied AIcited by 274
  13. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  14. 2024MM1: Methods, Analysis & Insights from Multimodal LLM Pre-trainingcited by 230
  15. 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
  16. 2024EMMA: End-to-End Multimodal Model for Autonomous Drivingcited by 226
  17. 2024RT-H: Action Hierarchies Using Languagecited by 204
  18. 2024PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMscited by 187
  19. 2024Building and better understanding vision-language models: insights and future directionscited by 169
  20. 2023Emu: Generative Pretraining in Multimodalitycited by 156
  21. 2024Multimodal ArXiv: A Dataset for Improving Scientific Comprehension of Large Vision-Language Modelscited by 138
  22. 2024Eagle: Exploring The Design Space for Multimodal LLMs with Mixture of Encoderscited by 137
  23. 2024Exploring the Reasoning Abilities of Multimodal Large Language Models (MLLMs): A Comprehensive Survey on Emerging Trends in Multimodal Reasoningcited by 136
  24. 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
  25. 2024ShapeLLM: Universal 3D Object Understanding for Embodied Interactioncited by 133
  26. 2023Large Language Models for Robotics: A Surveycited by 132
  27. 2023PaLI-3 Vision Language Models: Smaller, Faster, Strongercited by 125
  28. 2023MobileVLM : A Fast, Strong and Open Vision Language Assistant for Mobile Devicescited by 120
  29. 2025GraspVLA: a Grasping Foundation Model Pre-trained on Billion-scale Synthetic Action Datacited by 120
  30. 2024MagicLens: Self-Supervised Image Retrieval with Open-Ended Instructionscited by 119
  31. 2024xGen-MM (BLIP-3): A Family of Open Large Multimodal Modelscited by 116
  32. 2025GR-3 Technical Reportcited by 110
  33. 2023GeomVerse: A Systematic Evaluation of Large Models for Geometric Reasoningcited by 107
  34. 2026StarVLA: A Lego-like Codebase for Vision-Language-Action Model Developingcited by 103
  35. 2024ScreenAI: A Vision-Language Model for UI and Infographics Understandingcited by 97
  36. 2025A Survey on Vision-Language-Action Models: An Action Tokenization Perspectivecited by 96
  37. 2023Image Captioners Are Scalable Vision Learners Toocited by 89
  38. 2024Streaming Dense Video Captioningcited by 82

50 most cited of 254 All 254 (JSON)