Contextures: Representations from Contexts
Not on the map. Why?
Abstract (arXiv)
Despite the empirical success of foundation models, we do not have a systematic characterization of the representations that these models learn. In this paper, we establish the contexture theory. It shows that a large class of representation learning methods can be characterized as learning from the association between the input and a context variable. Specifically, we show that many popular methods aim to approximate the top-d singular functions of the expectation operator induced by the context, in which case we say that the representation learns the contexture. We demonstrate the generality of the contexture theory by proving that representation learning within various learning paradigms -- supervised, self-supervised, and manifold learning -- can all be studied from such a perspective. We also prove that the representations that learn the contexture are optimal on those tasks that are compatible with the context. One important implication of the contexture theory is that once the model is large enough to approximate the top singular functions, further scaling up the model size yields diminishing returns. Therefore, scaling is not all we need, and further improvement requires better contexts. To this end, we study how to evaluate the usefulness of a context without knowing the downstream tasks. We propose a metric and show by experiments that it correlates well with the actual performance of the encoder on many real datasets.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Builds on 42 of its 69 references that are in the map. First posted on arXiv 2 May 2025.
Builds on
42 of 69 references.
References · 42
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2022Training language models to follow instructions with human feedbackcited by 15,971
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
- 2014ImageNet Large Scale Visual Recognition Challengecited by 8,498
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,974
- 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
- 2020Scaling Laws for Neural Language Modelscited by 6,586
- 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
- 2022Emergent Abilities of Large Language Modelscited by 2,512
- 2013Deep Inside Convolutional Networks: Visualising Image Classification Models and Saliency Mapscited by 2,371
- 2013Visualizing and Understanding Convolutional Networkscited by 2,290
30 more
- 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,838
- 2019Similarity of Neural Network Representations Revisitedcited by 1,755
- 2016Understanding deep learning requires rethinking generalizationcited by 1,674
- 2021VICReg: Variance-Invariance-Covariance Regularization for Self-Supervised Learningcited by 1,057
- 2023Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecturecited by 1,039
- 2023Progress measures for grokking via mechanistic interpretabilitycited by 839
- 2022data2vec: A General Framework for Self-supervised Learning in Speech, Vision and Languagecited by 727
- 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 719
- 2014OpenML: networked science in machine learningcited by 526
- 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
- 2021Gradient Descent on Neural Networks Typically Occurs at the Edge of Stabilitycited by 427
- 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 378
- 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
- 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
- 2018Towards a Definition of Disentangled Representationscited by 296
- 2023A Cookbook of Self-Supervised Learningcited by 250
- 2017Emergence of Invariance and Disentanglement in Deep Representationscited by 215
- 2016What makes ImageNet good for transfer learning?cited by 192
- 2018Optimal approximation of continuous functions by very deep ReLU networkscited by 166
- 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
- 2022Contrastive and Non-Contrastive Self-Supervised Learning Recover Global and Local Spectral Embedding Methodscited by 149
- 2022Interpreting Language Models with Contrastive Explanationscited by 92
- 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
- 2022Imbalance Trouble: Revisiting Neural-Collapse Geometrycited by 80
- 2023Siamese Masked Autoencoderscited by 76
- 2023Multi-View Causal Representation Learning with Partial Observabilitycited by 74
- 2022Understanding Deep Contrastive Learning via Coordinate-wise Optimizationcited by 35
- 2023General Identifiability and Achievability for Causal Representation Learningcited by 34
- 2022Contrastive Learning Can Find An Optimal Basis For Approximately View-Invariant Functionscited by 32
- 2023Understanding Augmentation-based Self-Supervised Representation Learning via RKHS Approximation and Regressioncited by 20
Built on by
Citing papers per year: 2025: 1; 2026: 2.
3 papers.