mlsanityData to 7 Oct 2026

Simplicial Embeddings in Self-Supervised Learning and Downstream Classification

arXiv 2204.00616 ↗First posted 1 Apr 2022cs.LG cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Simplicial Embeddings (SEM) are representations learned through self-supervised learning (SSL), wherein a representation is projected into L simplices of V dimensions each using a softmax operation. This procedure conditions the representation onto a constrained space during pretraining and imparts an inductive bias for group sparsity. For downstream classification, we formally prove that the SEM representation leads to better generalization than an unnormalized representation. Furthermore, we empirically demonstrate that SSL methods trained with SEMs have improved generalization on natural image datasets such as CIFAR-100 and ImageNet. Finally, when used in a downstream classification task, we show that SEM features exhibit emergent semantic coherence where small groups of learned features are distinctly predictive of semantically-relevant classes.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 41 of its 77 references that are in the map. First posted on arXiv 1 Apr 2022.

Builds on

41 of 77 references.

References · 41

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  9. 2016Layer Normalizationcited by 5,579
  10. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  11. 2017Neural Discrete Representation Learningcited by 4,650
  12. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
29 more
  1. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  2. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  3. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  4. 2020Supervised Contrastive Learningcited by 2,803
  5. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  6. 2013Describing Textures in the Wildcited by 2,110
  7. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  8. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  11. 2020Graph Contrastive Learning with Augmentationscited by 1,203
  12. 2018Deep Clustering for Unsupervised Learning of Visual Featurescited by 1,020
  13. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  14. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  15. 2014Neural Turing Machinescited by 630
  16. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  17. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  18. 2019Revisiting Self-Supervised Visual Representation Learningcited by 250
  19. 2020Contrastive Learning of General-Purpose Audio Representationscited by 168
  20. 2019Adaptively Sparse Transformerscited by 154
  21. 2020Adversarial Robustness: From Self-Supervised Pre-Training to Fine-Tuningcited by 152
  22. 2021Solo-learn: A Library of Self-supervised Methods for Visual Representation Learningcited by 116
  23. 2020On Robustness and Transferability of Convolutional Neural Networkscited by 98
  24. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  25. 2015Sparse Overcomplete Word Vector Representationscited by 43
  26. 2021Compressive Visual Representationscited by 41
  27. 2021Discrete-Valued Neural Communicationcited by 39
  28. 2021Interpretable agent communication from scratch (with a generic visual processor emerging on the side)cited by 25
  29. 2021Escaping the Gradient Vanishing: Periodic Alternatives of Softmax in Attention Mechanismcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 7; 2024: 7; 2025: 5; 2026: 2.

23 papers.

Sort:

By area:

  1. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
  2. 2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 205
  3. 2023Know Your Self-supervised Learning: A Survey on Image-based Generative and Discriminative Trainingcited by 39
  4. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  5. 2023Improving Compositional Generalization Using Iterated Learning and Simplicial Embeddingscited by 15
  6. 2022Multi-CLS BERT: An Efficient Alternative to Traditional Ensemblingcited by 14
  7. 2024A Complexity-Based Theory of Compositionalitycited by 13
  8. 2024UniZero: Generalized and Efficient Planning with Scalable Latent World Modelscited by 12
  9. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  10. 2025Projection Head is Secretly an Information Bottleneckcited by 11
  11. 2025Learning Massively Multitask World Models for Continuous Controlcited by 11
  12. 2023Referential communication in heterogeneous communities of pre-trained visual deep networkscited by 10
11 more
  1. 2024Preventing Dimensional Collapse in Self-Supervised Learning via Orthogonality Regularizationcited by 9
  2. 2025Self-Supervised Evolution Operator Learning for High-Dimensional Dynamical Systemscited by 7
  3. 2026The Surprising Difficulty of Search in Model-Based Reinforcement Learningcited by 5
  4. 2024SPARO: Selective Attention for Robust and Compositional Transformer Encodings for Visioncited by 4
  5. 2022Quantitative Evidence on Overlooked Aspects of Enrollment Speaker Embeddings for Target Speaker Separationcited by 3
  6. 2024Taxes Are All You Need: Integration of Taxonomical Hierarchy Relationships into the Contrastive Losscited by 3
  7. 2023Discrete, compositional, and symbolic representations through attractor dynamicscited by 2
  8. 2024Faithful Differentiable Reasoning with Reshuffled Region-based Embeddingscited by 1
  9. 2025Compositional Discrete Latent Code for High Fidelity, Productive Diffusion Modelscited by 1
  10. 2023An Empirical Study of Self-supervised Learning with Wasserstein Distancecited by 0
  11. 2026Think Fast, Plan Selectively: Adaptive Deliberation for Efficient Data-Driven MPCcited by 0