mlsanityData to 7 Oct 2026

Neural Eigenfunctions Are Structured Representation Learners

arXiv 2210.12637 ↗First posted 23 Oct 2022cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

This paper introduces a structured, adaptive-length deep representation called Neural Eigenmap. Unlike prior spectral methods such as Laplacian Eigenmap that operate in a nonparametric manner, Neural Eigenmap leverages NeuralEF to parametrically model eigenfunctions using a neural network. We show that, when the eigenfunction is derived from positive relations in a data augmentation setup, applying NeuralEF results in an objective function that resembles those of popular self-supervised learning methods, with an additional symmetry-breaking property that leads to structured representations where features are ordered by importance. We demonstrate using such representations as adaptive-length codes in image retrieval systems. By truncation according to feature importance, our method requires up to 16× shorter representation length than leading self-supervised learning ones to achieve similar retrieval performance. We further apply our method to graph data and report strong results on a node representation learning benchmark with more than one million nodes.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 57 of its 91 references that are in the map. First posted on arXiv 23 Oct 2022.

Builds on

57 of 91 references.

References · 57

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  6. 2020Language Models are Few-Shot Learnerscited by 32,199
  7. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  8. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  9. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  10. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  11. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  12. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
38 more
  1. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  2. 2017Mask R-CNNcited by 7,259
  3. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  4. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  5. 2017Inductive Representation Learning on Large Graphscited by 6,284
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  10. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  11. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  12. 2020Improved Baselines with Momentum Contrastive Learningcited by 2,282
  13. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  14. 2016node2vec: Scalable Feature Learning for Networkscited by 1,913
  15. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
  16. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  17. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  18. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  19. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  20. 2016Colorful Image Colorizationcited by 1,113
  21. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  22. 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
  23. 2021iBOT: Image BERT Pre-Training with Online Tokenizercited by 891
  24. 2015Automatic differentiation in machine learning: a surveycited by 846
  25. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  26. 2017Large Batch Training of Convolutional Networkscited by 529
  27. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  28. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  29. 2022Masked Siamese Networks for Label-Efficient Learningcited by 301
  30. 2022Matryoshka Representation Learningcited by 268
  31. 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
  32. 2020Contrastive learning, multi-view redundancy, and linear modelscited by 146
  33. 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
  34. 2016Learning Features by Watching Objects Movecited by 138
  35. 2021Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implicationscited by 128
  36. 2022VICRegL: Self-Supervised Learning of Local Visual Featurescited by 118
  37. 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
  38. 2022Pushing the limits of self-supervised ResNets: Can we outperform supervised learning without labels on ImageNet?cited by 67

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 1; 2025: 2; 2026: 3.

16 papers.

Sort:

By area:

  1. 2023Flow Matching on General Geometriescited by 260
  2. 2023Improved Operator Learning by Orthogonal Attentioncited by 58
  3. 2023Deep Laplacian-based Options for Temporally-Extended Explorationcited by 28
  4. 2023Contrastive Learning Is Spectral Clustering On Similarity Graphcited by 24
  5. 2023The Edge of Orthogonality: A Simple View of What Makes BYOL Tickcited by 15
  6. 2022Spectral Representation Learning for Conditional Moment Modelscited by 12
  7. 2024Operator SVD with Neural Networks via Nested Low-Rank Approximationcited by 12
  8. 2023The Galerkin method beats Graph-Based Approaches for Spectral Algorithmscited by 6
  9. 2023Harnessing small projectors and multiple views for efficient vision pretrainingcited by 5
  10. 2023Learning Neural Eigenfunctions for Unsupervised Semantic Segmentationcited by 4
  11. 2026Intrinsic Green's Learning: Supervised Learning on Manifolds via Inverse PDEcited by 1
  12. 2023Contrastive Learning as Kernel Approximationcited by 0
4 more
  1. 2025Revisiting Orbital Minimization Method for Neural Operator Decompositioncited by 0
  2. 2025Understanding InfoNCE: Transition Probability Matrix Induced Feature Clusteringcited by 0
  3. 2026From Complex Dynamics to DynFormer: Rethinking Transformers for PDEscited by 0
  4. 2026Revisiting Spectral Representations in Generative Diffusion Modelscited by 0