Eigenfunction Extraction for Ordered Representation Learning
Not on the map. Why?
Abstract (arXiv)
Recent advances in representation learning reveal that widely used objectives, such as contrastive and non-contrastive, implicitly perform spectral decomposition of a contextual kernel, induced by the relationship between inputs and their contexts. Yet, these methods recover only the linear span of top eigenfunctions of the kernel, whereas exact spectral decomposition is essential for understanding feature ordering and importance. In this work, we propose a general framework to extract ordered and identifiable eigenfunctions, based on modular building blocks designed to satisfy key desiderata, including compatibility with the contextual kernel and scalability to modern settings. We then show how two main methodological paradigms, low-rank approximation and Rayleigh quotient optimization, align with this framework for eigenfunction extraction. Finally, we validate our approach on synthetic kernels and demonstrate on real-world image datasets that the recovered eigenvalues act as effective importance scores for feature selection, enabling principled efficiency-accuracy tradeoffs via adaptive-dimensional representations.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 85 of its 108 references that are in the map. First posted on arXiv 28 Oct 2025.
Builds on
85 of 108 references.
References · 85
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
- 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
38 more
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
- 2020Exploring Simple Siamese Representation Learningcited by 2,853
- 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
- 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
- 2021Convex optimizationcited by 2,479
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
- 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
- 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
- 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
- 2025SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Featurescited by 1,127
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,013
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 982
- 2016Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphscited by 853
- 2021Florence: A New Foundation Model for Computer Visioncited by 802
- 2025V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planningcited by 738
- 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
- 2019On Mutual Information Maximization for Representation Learningcited by 315
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
- 2022Matryoshka Representation Learningcited by 268
- 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
- 2023A Cookbook of Self-Supervised Learningcited by 248
- 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
- 2020Predicting What You Already Know Helps: Provable Self-Supervised Learningcited by 152
- 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 146
- 2022RankMe: Assessing the downstream performance of pretrained self-supervised representations by their rankcited by 145
- 2022Interventional Causal Representation Learningcited by 140
- 2025Learning Multi-Level Features with Matryoshka Sparse Autoencoderscited by 136
- 2023Hyperbolic Image-Text Representationscited by 132
- 2025EmbeddingGemma: Powerful and Lightweight Text Representationscited by 128
- 2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
- 2022On the duality between contrastive and non-contrastive self-supervised learningcited by 107
- 2022Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlapcited by 105
- 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
- 2024Matryoshka Multimodal Modelscited by 82
- 2024On the Origins of Linear Representations in Large Language Modelscited by 76
- 2023Matryoshka Diffusion Modelscited by 70
50 most cited of 85 All 85 (JSON)
Built on by
Not cited yet.