mlsanityData to 14 Oct 2026

Efficiency for Free: Ideal Data Are Transportable Representations

arXiv 2405.14669 ↗First posted 23 May 2024cs.LG cs.AIcited by 2

Not on the map. Why?

Abstract (arXiv)

Data, the seminal opportunity and challenge in modern machine learning, currently constrains the scalability of representation learning and impedes the pace of model evolution. In this work, we investigate the efficiency properties of data from both optimization and generalization perspectives. Our theoretical and empirical analysis reveals an unexpected finding: for a given task, utilizing a publicly available, task- and architecture-agnostic model (referred to as the `prior model' in this paper) can effectively produce efficient data. Building on this insight, we propose the Representation Learning Accelerator (algopt), which promotes the formation and utilization of efficient data, thereby accelerating representation learning. Utilizing a ResNet-18 pre-trained on CIFAR-10 as a prior model to inform ResNet-50 training on ImageNet-1K reduces computational costs by 50% while maintaining the same accuracy as the model trained with the original BYOL, which requires 100% cost. Our code is available at: urlhttps://github.com/LINs-lab/ReLA.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 60 of its 82 references that are in the map. First posted on arXiv 23 May 2024.

Builds on

60 of 82 references.

References · 60

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  8. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  10. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,968
  11. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  12. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,795
38 more
  1. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,866
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,857
  4. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  5. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,287
  6. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
  7. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,616
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  9. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,254
  10. 2017Visualizing the Loss Landscape of Neural Netscited by 1,130
  11. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
  12. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  13. 2020Prototypical Contrastive Learning of Unsupervised Representationscited by 640
  14. 2022Beyond neural scaling laws: beating power law scaling via data pruningcited by 528
  15. 2020Dataset Condensation with Gradient Matchingcited by 516
  16. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  17. 2022Dataset Distillation by Matching Training Trajectoriescited by 442
  18. 2020Hard Negative Mixing for Contrastive Learningcited by 402
  19. 2021Dataset Condensation with Distribution Matchingcited by 365
  20. 2017A Survey of Model Compression and Acceleration for Deep Neural Networkscited by 301
  21. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  22. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  23. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 256
  24. 2023Sequential Modeling Enables Scalable Learning for Large Vision Modelscited by 212
  25. 2018Don't Use Large Mini-Batches, Use Local SGDcited by 202
  26. 2019Scaling and Benchmarking Self-Supervised Visual Representation Learningcited by 201
  27. 2022Dataset Condensation via Efficient Synthetic-Data Parameterizationcited by 188
  28. 2022Scaling Up Dataset Distillation to ImageNet-1K with Constant Memorycited by 181
  29. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  30. 2023Squeeze, Recover and Relabel: Dataset Condensation at ImageNet Scale From A New Perspectivecited by 138
  31. 2023Dataset Distillation: A Comprehensive Reviewcited by 127
  32. 2023Improved Distribution Matching for Dataset Condensationcited by 124
  33. 2023Towards Lossless Dataset Distillation via Difficulty-Aligned Trajectory Matchingcited by 118
  34. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  35. 2023On the Diversity and Realism of Distilled Dataset: An Efficient Dataset Distillation Paradigmcited by 109
  36. 2020BYOL works even without batch statisticscited by 95
  37. 2023DREAM: Efficient Dataset Distillation by Representative Matchingcited by 94
  38. 2020AdCo: Adversarial Contrast for Efficient Learning of Unsupervised Representations from Self-Trained Negative Adversariescited by 90

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Collaborative Unlabeled Data Optimizationcited by 0
  2. 2025Equally Critical: Samples, Targets, and Their Mappings in Datasetscited by 0