mlsanityData to 14 Oct 2026

Towards artificial general intelligence via a multimodal foundation model

arXiv 2110.14378 ↗First posted 27 Oct 2021cs.AIcited by 112

Not on the map. Why?

Abstract (arXiv)

The fundamental goal of artificial intelligence (AI) is to mimic the core cognitive activities of human. Despite tremendous success in the AI research, most of existing methods have only single-cognitive ability. To overcome this limitation and take a solid step towards artificial general intelligence (AGI), we develop a foundation model pre-trained with huge multimodal data, which can be quickly adapted for various downstream cognitive tasks. To achieve this goal, we propose to pre-train our foundation model by self-supervised learning with weak semantic correlation data crawled from the Internet and show that promising results can be obtained on a wide range of downstream tasks. Particularly, with the developed model-interpretability tools, we demonstrate that strong imagination ability is now possessed by our foundation model. We believe that our work makes a transformative stride towards AGI, from our common practice of "weak or narrow AI" to that of "strong or generalized AI".

Key facts. Cited by 112 papers in the map (data through 14 Oct 2026). Builds on 43 of its 68 references that are in the map. First posted on arXiv 27 Oct 2021.

Builds on

43 of 68 references.

References · 43

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  6. 2020Language Models are Few-Shot Learnerscited by 32,419
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  12. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
31 more
  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  2. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2017Neural Discrete Representation Learningcited by 4,712
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  9. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,253
  10. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  11. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  12. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,312
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,296
  14. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  15. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,214
  16. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  17. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  18. 2021Scaling Vision with Sparse Mixture of Expertscited by 747
  19. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  20. 2016AID: A Benchmark Dataset for Performance Evaluation of Aerial Scene Classificationcited by 347
  21. 2017A simple neural network module for relational reasoningcited by 346
  22. 2015Visual7W: Grounded Question Answering in Imagescited by 318
  23. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 266
  24. 2020Counterfactual VQA: A Cause-Effect Look at Language Biascited by 266
  25. 2021ZeRO-Infinity: Breaking the GPU Memory Wall for Extreme Scale Deep Learningcited by 256
  26. 2019Local Aggregation for Unsupervised Learning of Visual Embeddingscited by 204
  27. 2021Similarity Reasoning and Filtration for Image-Text Matchingcited by 121
  28. 2020IMRAM: Iterative Matching with Recurrent Attention Memory for Cross-Modal Image-Text Retrievalcited by 115
  29. 2021M6: A Chinese Multimodal Pretrainercited by 91
  30. 2017AI Challenger : A Large-scale Dataset for Going Deeper in Image Understandingcited by 73
  31. 2018Zero and Few Shot Learning with Semantic Feature Synthesis and Competitive Learningcited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 7; 2023: 31; 2024: 35; 2025: 27; 2026: 12.

112 papers.

Sort:

By area:

  1. 2023A Survey of Large Language Models for Healthcare: from Data, Technology, and Applications to Accountability and Ethicscited by 163
  2. 2022A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Languagecited by 131
  3. 2023Graph Foundation Models: Concepts, Opportunities and Challengescited by 103
  4. 2025Universal Actions for Enhanced Embodied Foundation Modelscited by 82
  5. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  6. 2024Multimodal Alignment and Fusion: A Surveycited by 74
  7. 2024HyperSIGMA: Hyperspectral Intelligence Comprehension Foundation Modelcited by 73
  8. 2022Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Featurescited by 63
  9. 2024LLM3:Large Language Model-based Task and Motion Planning with Motion Failure Reasoningcited by 63
  10. 2023How Robust is GPT-3.5 to Predecessors? A Comprehensive Study on Language Understanding Taskscited by 58
  11. 2024Quantifying and Mitigating Unimodal Biases in Multimodal Large Language Models: A Causal Perspectivecited by 39
  12. 2023FAME-ViL: Multi-Tasking Vision-Language Model for Heterogeneous Fashion Taskscited by 38
38 more
  1. 2025BadVLA: Towards Backdoor Attacks on Vision-Language-Action Models via Objective-Decoupled Optimizationcited by 38
  2. 2023Emergent Modularity in Pre-trained Transformerscited by 37
  3. 2024Benchmarks as Microscopes: A Call for Model Metrologycited by 35
  4. 2025Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoningcited by 33
  5. 2024Review of multimodal machine learning approaches in healthcarecited by 31
  6. 2023CLIP4STR: A Simple Baseline for Scene Text Recognition with Pre-trained Vision-Language Modelcited by 30
  7. 2023The Shaky Foundations of Clinical Foundation Models: A Survey of Large Language Models and Foundation Models for EMRscited by 29
  8. 2024Deep Generative Data Assimilation in Multimodal Settingcited by 29
  9. 2023Multimodality Representation Learning: A Survey on Evolution, Pretraining and Its Applicationscited by 27
  10. 2024Watermarking Techniques for Large Language Models: A Surveycited by 23
  11. 2023Machine Psychologycited by 22
  12. 2022CCMB: A Large-scale Chinese Cross-modal Benchmarkcited by 21
  13. 2024When Large Language Models Meet Evolutionary Algorithms: Potential Enhancements and Challengescited by 20
  14. 2024Fairness and Bias in Multimodal AI: A Surveycited by 20
  15. 2022WeLM: A Well-Read Pre-trained Language Model for Chinesecited by 19
  16. 2024An overview of domain-specific foundation model: key technologies, applications and challengescited by 19
  17. 2023From Google Gemini to OpenAI Q* (Q-Star): A Survey of Reshaping the Generative Artificial Intelligence (AI) Research Landscapecited by 18
  18. 2023Fine-Tuning Games: Bargaining and Adaptation for General-Purpose Modelscited by 17
  19. 2025I2-NeRF: Learning Neural Radiance Fields Under Physically-Grounded Media Interactionscited by 17
  20. 2023Influence of External Information on Large Language Models Mirrors Social Cognitive Patternscited by 16
  21. 2024Taxonomy-Guided Zero-Shot Recommendations with LLMscited by 15
  22. 2025SafeMobile: Chain-level Jailbreak Detection and Automated Evaluation for Multimodal Mobile Agentscited by 15
  23. 2023Towards Artificial General Intelligence (AGI) in the Internet of Things (IoT): Opportunities and Challengescited by 14
  24. 2024The Road to Artificial SuperIntelligence: A Comprehensive Survey of Superalignmentcited by 13
  25. 2025Aligning Multimodal Representations through an Information Bottleneckcited by 13
  26. 2024LLM-driven Imitation of Subrational Behavior : Illusion or Reality?cited by 12
  27. 2022LGDN: Language-Guided Denoising Network for Video-Language Modelingcited by 11
  28. 2023Towards Medical Artificial General Intelligence via Knowledge-Enhanced Multimodal Pretrainingcited by 11
  29. 2025Large Wireless Localization Model (LWLM): A Foundation Model for Positioning in 6G Networkscited by 11
  30. 2022Multimodal foundation models are better simulators of the human braincited by 10
  31. 2025Geospatial Foundation Models to Enable Progress on Sustainable Development Goalscited by 10
  32. 2023Active Self-Supervised Learning: A Few Low-Cost Relationships Are All You Needcited by 9
  33. 2024AutoTRIZ: Automating Engineering Innovation with TRIZ and Large Language Modelscited by 9
  34. 2023Vision + Language Applications: A Surveycited by 8
  35. 2023Towards a Psychological Generalist AI: A Survey of Current Applications of Large Language Models and Future Prospectscited by 8
  36. 2024Enhancing Adversarial Robustness of Vision-Language Models through Low-Rank Adaptationcited by 8
  37. 2024Tissue Concepts: supervised foundation models in computational pathologycited by 8
  38. 2024Optical Generative Modelscited by 8

50 most cited of 112 All 112 (JSON)