mlsanityData to 14 Oct 2026

Empirical Evaluation and Theoretical Analysis for Representation Learning: A Survey

arXiv 2204.08226 ↗First posted 18 Apr 2022cs.LG cs.CVcited by 5

Not on the map. Why?

Abstract (arXiv)

Representation learning enables us to automatically extract generic feature representations from a dataset to solve another machine learning task. Recently, extracted feature representations by a representation learning algorithm and a simple predictor have exhibited state-of-the-art performance on several machine learning tasks. Despite its remarkable progress, there exist various ways to evaluate representation learning algorithms depending on the application because of the flexibility of representation learning. To understand the current representation learning, we review evaluation methods of representation learning algorithms and theoretical analyses. On the basis of our evaluation survey, we also discuss the future direction of representation learning. Note that this survey is the extended version of Nozawa and Sato (2022).

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 137 of its 208 references that are in the map. First posted on arXiv 18 Apr 2022.

Builds on

137 of 208 references.

References · 137

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  7. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  8. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  2. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  3. 2014Fully Convolutional Networks for Semantic Segmentationcited by 4,859
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  5. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,375
  6. 2013Auto-Encoding Variational Bayescited by 4,053
  7. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,474
  8. 2015Fast R-CNNcited by 2,933
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  11. 2018Deep contextualized word representationscited by 2,290
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  13. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,181
  14. 2013Rich feature hierarchies for accurate object detection and semantic segmentationcited by 2,162
  15. 2016node2vec: Scalable Feature Learning for Networkscited by 1,918
  16. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  17. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,729
  18. 2014FitNets: Hints for Thin Deep Netscited by 1,653
  19. 2020Knowledge Distillation: A Surveycited by 1,603
  20. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,591
  21. 2014DeepWalk: Online Learning of Social Representationscited by 1,547
  22. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  23. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,382
  24. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,273
  25. 2014How transferable are features in deep neural networks?cited by 1,225
  26. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,066
  27. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 919
  28. 2021GLaM: Efficient Scaling of Language Models with Mixture-of-Expertscited by 897
  29. 2020What Makes for Good Views for Contrastive Learning?cited by 846
  30. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  31. 2019Self-Supervised Learning of Pretext-Invariant Representationscited by 808
  32. 2019Big Transfer (BiT): General Visual Representation Learningcited by 769
  33. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  34. 2019Learning Representations by Maximizing Mutual Information Across Viewscited by 734
  35. 2019A Closer Look at Few-shot Classificationcited by 684
  36. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  37. 2019Self-supervised Visual Feature Learning with Deep Neural Networks: A Surveycited by 657
  38. 2020A Survey on Knowledge Graphs: Representation, Acquisition and Applicationscited by 643

50 most cited of 137 All 137 (JSON)

Built on by

Cited by year

Citing papers per year: 2022: 1; 2023: 2; 2024: 1; 2025: 1.

5 papers.

  1. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 25
  2. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationaritycited by 5
  3. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  4. 2023Evaluating Representations with Readout Model Switchingcited by 2
  5. 2025Self-Calibrating BCIs: Ranking and Recovery of Mental Targets Without Labelscited by 1