mlsanityData to 14 Oct 2026

Big Learning

arXiv 2207.03899 ↗First posted 8 Jul 2022cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent advances in big/foundation models reveal a promising path for deep learning, where the roadmap steadily moves from big data to big models to (the newly-introduced) big learning. Specifically, the big learning exhaustively exploits the information inherent in its large-scale complete/incomplete training data, by simultaneously modeling many/all joint/conditional/marginal data distributions across potentially diverse domains, with one universal foundation model. We reveal that big learning (i) underlies most existing foundation models, (ii) is equipped with extraordinary flexibilities for complete/incomplete training data and trustworthy data tasks, (iii) is capable of delivering all joint/conditional/marginal data capabilities with one universal model, and (iv) unifies conventional machine learning paradigms and enables their flexible cooperations, manifested as a universal learning paradigm. Diverse experiments are carried out to validate the effectiveness of the presented big learning.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 52 of its 76 references that are in the map. First posted on arXiv 8 Jul 2022.

Builds on

52 of 76 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020Language Models are Few-Shot Learnerscited by 32,419
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  6. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  8. 2022Training language models to follow instructions with human feedbackcited by 15,971
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  10. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  11. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  12. 2021Evaluating Large Language Models Trained on Codecited by 8,320
38 more
  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  2. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,288
  9. 2021On the Opportunities and Risks of Foundation Modelscited by 4,033
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,588
  11. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  13. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  14. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,626
  15. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,375
  16. 2016Density estimation using Real NVPcited by 2,042
  17. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  18. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,545
  19. 2020Training Generative Adversarial Networks with Limited Datacited by 1,186
  20. 2021Alias-Free Generative Adversarial Networkscited by 1,184
  21. 2014NICE: Non-linear Independent Components Estimationcited by 1,058
  22. 2021Florence: A New Foundation Model for Computer Visioncited by 804
  23. 2021Going deeper with Image Transformerscited by 764
  24. 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
  25. 2018Which Training Methods for GANs do actually Converge?cited by 679
  26. 2021Masked Feature Prediction for Self-Supervised Visual Pre-Trainingcited by 576
  27. 2021Intriguing Properties of Vision Transformerscited by 446
  28. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  29. 2021Escaping the Big Data Paradigm with Compact Transformerscited by 264
  30. 2019Diagnosing and Enhancing VAE Modelscited by 193
  31. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  32. 2021TransGAN: Two Pure Transformers Can Make One Strong GAN, and That Can Scale Upcited by 162
  33. 2021Pretrained Transformers as Universal Computation Enginescited by 162
  34. 2021StyleSwin: Transformer-based GAN for High-resolution Image Generationcited by 152
  35. 2021ViTGAN: Training GANs with Vision Transformerscited by 110
  36. 2021Improved Transformer for High-Resolution GANscited by 59
  37. 2021LIME: Learning Inductive Bias for Primitives of Mathematical Reasoningcited by 46
  38. 2022Towards Data-Efficient Detection Transformerscited by 34

50 most cited of 52 All 52 (JSON)

Built on by

Not cited yet.