mlsanityData to 7 Oct 2026

M6: A Chinese Multimodal Pretrainer

arXiv 2103.00823 ↗First posted 1 Mar 2021cs.CLcited by 91

Not on the map. Why?

Abstract (arXiv)

In this work, we construct the largest dataset for multimodal pretraining in Chinese, which consists of over 1.9TB images and 292GB texts that cover a wide range of domains. We propose a cross-modal pretraining method called M6, referring to Multi-Modality to Multi-Modality Multitask Mega-transformer, for unified pretraining on the data of single modality and multiple modalities. We scale the model size up to 10 billion and 100 billion parameters, and build the largest pretrained model in Chinese. We apply the model to a series of downstream applications, and demonstrate its outstanding performance in comparison with strong baselines. Furthermore, we specifically design a downstream task of text-guided image generation, and show that the finetuned M6 can create high-quality images with high resolution and abundant details.

Key facts. Cited by 91 papers in the map (data through 7 Oct 2026). Builds on 44 of its 51 references that are in the map. First posted on arXiv 1 Mar 2021.

Builds on

44 of 51 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2017Neural Discrete Representation Learningcited by 4,650
  11. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  12. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
32 more
  1. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  3. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  4. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  5. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  9. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  10. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  11. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  12. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  13. 2019VisualBERT: A Simple and Performant Baseline for Vision and Languagecited by 1,291
  14. 2020Oscar: Object-Semantics Aligned Pre-training for Vision-Language Taskscited by 1,213
  15. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  16. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
  17. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  18. 2019Unified Vision-Language Pre-Training for Image Captioning and VQAcited by 494
  19. 2019Unicoder-VL: A Universal Encoder for Vision and Language by Cross-modal Pre-trainingcited by 451
  20. 2019MASS: Masked Sequence to Sequence Pre-training for Language Generationcited by 345
  21. 2020Large-Scale Adversarial Training for Vision-and-Language Representation Learningcited by 327
  22. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  23. 201912-in-1: Multi-Task Vision and Language Representation Learningcited by 278
  24. 2020Revisiting Pre-Trained Models for Chinese Natural Language Processingcited by 262
  25. 2020ERNIE-ViL: Knowledge Enhanced Vision-Language Representations Through Scene Graphcited by 211
  26. 2020UniLMv2: Pseudo-Masked Language Models for Unified Language Model Pre-Trainingcited by 204
  27. 2019StructBERT: Incorporating Language Structures into Pre-training for Deep Language Understandingcited by 108
  28. 2020ConvBERT: Improving BERT with Span-based Dynamic Convolutioncited by 88
  29. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  30. 2020FashionBERT: Text and Image Matching with Adaptive Loss for Cross-modal Retrievalcited by 55
  31. 2020InterBERT: Vision-and-Language Interaction for Multi-modal Pretrainingcited by 48
  32. 2020CLUECorpus2020: A Large-scale Chinese Corpus for Pre-training Language Modelcited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 31; 2023: 20; 2024: 13; 2025: 4; 2026: 1.

91 papers.

Sort:

By area:

  1. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  2. 2024DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Modelscited by 834
  3. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  4. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 742
  5. 2021CogView: Mastering Text-to-Image Generation via Transformerscited by 691
  6. 2021FILIP: Fine-grained Interactive Language-Image Pre-Trainingcited by 626
  7. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 557
  8. 2021A Survey of Transformerscited by 399
  9. 2023Uni-ControlNet: All-in-One Control to Text-to-Image Diffusion Modelscited by 358
  10. 2024SWIFT:A Scalable lightWeight Infrastructure for Fine-Tuningcited by 345
  11. 2021ERNIE 3.0: Large-scale Knowledge Enhanced Pre-training for Language Understanding and Generationcited by 258
  12. 2023Large-scale Multi-Modal Pre-trained Models: A Comprehensive Surveycited by 153
38 more
  1. 2021PanGu-α: Large-scale Autoregressive Pretrained Chinese Language Models with Auto-parallel Computationcited by 134
  2. 2023ONE-PEACE: Exploring One General Representation Model Toward Unlimited Modalitiescited by 117
  3. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  4. 2022Wukong: A 100 Million Large-scale Chinese Cross-modal Pre-training Benchmarkcited by 114
  5. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 113
  6. 2021Towards artificial general intelligence via a multimodal foundation modelcited by 112
  7. 2022Chinese CLIP: Contrastive Vision-Language Pretraining in Chinesecited by 108
  8. 2021Scale Efficiently: Insights from Pre-training and Fine-tuning Transformerscited by 105
  9. 2023GALIP: Generative Adversarial CLIPs for Text-to-Image Synthesiscited by 84
  10. 2022Plausible May Not Be Faithful: Probing Object Hallucination in Vision-Language Pre-trainingcited by 71
  11. 2022Enabling Multimodal Generation on CLIP via Vision-Language Knowledge Distillationcited by 70
  12. 2022Open-world Semantic Segmentation via Contrasting and Clustering Vision-Language Embeddingcited by 67
  13. 2022Improved Vector Quantized Diffusion Modelscited by 62
  14. 2025Vchitect-2.0: Parallel Transformer for Scaling Up Video Diffusion Modelscited by 58
  15. 2021A Survey on Green Deep Learningcited by 52
  16. 2023FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placementcited by 52
  17. 2022Modeling Image Composition for Complex Scene Generationcited by 43
  18. 2022You Need Multiple Exiting: Dynamic Early Exiting for Accelerating Unified Vision Language Modelcited by 42
  19. 2024Autoregressive Models in Vision: A Surveycited by 39
  20. 2021Edge-Cloud Polarization and Collaboration: A Comprehensive Survey for AIcited by 38
  21. 2021On the Limitations of Multimodal VAEscited by 37
  22. 2021M6-UFC: Unifying Multi-Modal Controls for Conditional Image Synthesis via Non-Autoregressive Generative Transformerscited by 36
  23. 2021Yuan 1.0: Large-Scale Pre-trained Language Model in Zero-Shot and Few-Shot Learningcited by 35
  24. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  25. 2021M6-10T: A Sharing-Delinking Paradigm for Efficient Multi-Trillion Parameter Pretrainingcited by 31
  26. 2022Extreme Generative Image Compression by Learning Text Embedding from Diffusion Modelscited by 30
  27. 2022DiVAE: Photorealistic Images Synthesis with Denoising Diffusion Decodercited by 27
  28. 2021M5Product: Self-harmonized Contrastive Learning for E-commercial Multi-modal Pretrainingcited by 26
  29. 2022A Survey on Dynamic Neural Networks for Natural Language Processingcited by 24
  30. 2024OmniControlNet: Dual-stage Integration for Conditional Image Generationcited by 24
  31. 2021Understanding Chinese Video and Language via Contrastive Multimodal Pre-Trainingcited by 23
  32. 2021M6-T: Exploring Sparse Expert Models and Beyondcited by 22
  33. 2021Learning Relation Alignment for Calibrated Cross-modal Retrievalcited by 21
  34. 2022CCMB: A Large-scale Chinese Cross-modal Benchmarkcited by 21
  35. 2025Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgettingcited by 16
  36. 2023CatVersion: Concatenating Embeddings for Diffusion-Based Text-to-Image Personalizationcited by 15
  37. 2022Masked Vision-Language Transformer in Fashioncited by 14
  38. 2022Fine-grained Visual-Text Prompt-Driven Self-Training for Open-Vocabulary Object Detectioncited by 14

50 most cited of 91 All 91 (JSON)