mlsanityData to 7 Oct 2026

Deep Learning of Representations: Looking Forward

arXiv 1305.0445 ↗First posted 2 May 2013cs.LGcited by 84

Not on the map. Why?

Abstract (arXiv)

Deep learning research aims at discovering learning algorithms that discover multiple levels of distributed representations, with higher levels representing more abstract concepts. Although the study of deep learning has already led to impressive theoretical results, learning algorithms and breakthrough experiments, several challenges lie ahead. This paper proposes to examine some of these challenges, centering on the questions of scaling deep learning algorithms to much larger models and datasets, reducing optimization difficulties due to ill-conditioning or local minima, designing more efficient and powerful inference and sampling procedures, and learning to disentangle the factors of variation underlying the observed data. It also proposes a few forward-looking research directions aimed at overcoming these challenges.

Key facts. Cited by 84 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 May 2013.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 1; 2021: 21; 2022: 13; 2023: 12; 2024: 23; 2025: 11; 2026: 3.

84 papers.

Sort:

By area:

  1. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  2. 2024Mixture-of-Depths: Dynamically allocating compute in transformer-based language modelscited by 242
  3. 2024Improving Dictionary Learning with Gated Sparse Autoencoderscited by 171
  4. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  5. 2024Interpreting CLIP with Sparse Linear Concept Embeddings (SpLiCE)cited by 116
  6. 2022M3ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-designcited by 97
  7. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  8. 2021A Practical Survey on Faster and Lighter Transformerscited by 73
  9. 2024Mixture of A Million Expertscited by 70
  10. 2024ReLU2 Wins: Discovering Efficient Activation Functions for Sparse LLMscited by 66
  11. 2024Scaling Diffusion Transformers to 16 Billion Parameterscited by 65
  12. 2021Learning Task Informed Abstractionscited by 58
38 more
  1. 2021Go Wider Instead of Deepercited by 56
  2. 2021CPM-2: Large-scale Cost-effective Pre-trained Language Modelscited by 52
  3. 2024Training-Free Activation Sparsity in Large Language Modelscited by 51
  4. 2024CiteME: Can Language Models Accurately Cite Scientific Claims?cited by 45
  5. 2023Disentanglement via Latent Quantizationcited by 40
  6. 2024Turbo Sparse: Achieving LLM SOTA Performance with Minimal Activated Parameterscited by 37
  7. 2021Orthogonal Jacobian Regularization for Unsupervised Disentanglement in Image Generationcited by 35
  8. 2022Is a Modular Architecture Enough?cited by 35
  9. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  10. 2024Mixture of Nested Experts: Adaptive Processing of Visual Tokenscited by 25
  11. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competitioncited by 23
  12. 2021Entity Linking Meets Deep Learning: Techniques and Solutionscited by 21
  13. 2024SA-DVAE: Improving Zero-Shot Skeleton-Based Action Recognition by Disentangled Variational Autoencoderscited by 19
  14. 2023Towards Convergence Rates for Parameter Estimation in Gaussian-gated Mixture of Expertscited by 17
  15. 2024Memory Mosaicscited by 16
  16. 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 15
  17. 2022Spatial Mixture-of-Expertscited by 14
  18. 2024Tripod: Three Complementary Inductive Biases for Disentangled Representation Learningcited by 13
  19. 2021Benchmarks, Algorithms, and Metrics for Hierarchical Disentanglementcited by 12
  20. 2025Towards Interpretable Deep Generative Models via Causal Representation Learningcited by 12
  21. 2021Training a Resilient Q-Network against Observational Interferencecited by 9
  22. 2025Subspace-Boosted Model Mergingcited by 9
  23. 2021UnProjection: Leveraging Inverse-Projections for Visual Analytics of High-Dimensional Datacited by 8
  24. 2022Deep Double Descent via Smooth Interpolationcited by 8
  25. 2022A Survey of Neural Treescited by 7
  26. 2020Non-asymptotic oracle inequalities for the Lasso in high-dimensional mixture of expertscited by 6
  27. 2022Towards Robust Unsupervised Disentanglement of Sequential Data -- A Case Study Using Music Audiocited by 6
  28. 2023Inversion dynamics of class manifolds in deep learning reveals tradeoffs underlying generalisationcited by 6
  29. 2023Learning Disentangled Semantic Spaces of Explanations via Invertible Neural Networkscited by 6
  30. 2023Adaptive Computation Modules: Granular Conditional Computation For Efficient Inferencecited by 6
  31. 2024MerRec: A Large-scale Multipurpose Mercari Dataset for Consumer-to-Consumer Recommendation Systemscited by 6
  32. 2022Quasi-symbolic Semantic Geometry over Transformer-based Variational AutoEncodercited by 5
  33. 2023Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulationcited by 5
  34. 2025Teach Old SAEs New Domain Tricks with Boostingcited by 5
  35. 2021On the relationship between disentanglement and multi-task learningcited by 4
  36. 2021Encoding Causal Macrovariablescited by 4
  37. 2022Doge Tickets: Uncovering Domain-general Language Models by Playing Lottery Ticketscited by 4
  38. 2023Neuro-Causal Factor Analysiscited by 4

50 most cited of 84 All 84 (JSON)