# On Universal Features for High-Dimensional Learning and Inference

arXiv: https://arxiv.org/abs/1911.09105 · HTML: https://mlsanity.com/paper/1911.09105/

## Key facts

- Cited by 20 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 20 Nov 2019.

## Built on by (most cited first, 20 of 20)

- [An Information-Theoretic Approach to Transferability in Task Transfer Learning](https://mlsanity.com/paper/2212.10082/) (2022)
- [PACTran: PAC-Bayesian Metrics for Estimating the Transferability of Pretrained Models to Classification Tasks](https://mlsanity.com/paper/2203.05126/) (2022)
- [Operator SVD with Neural Networks via Nested Low-Rank Approximation](https://mlsanity.com/paper/2402.03655/) (2024)
- [H-ensemble: An Information Theoretic Approach to Reliable Few-Shot Multi-Source-Free Transfer](https://mlsanity.com/paper/2312.12489/) (2023)
- [Timely Communications for Remote Inference](https://mlsanity.com/paper/2404.16281/) (2024)
- [The Age of Correlated Features in Supervised Learning based Forecasting](https://mlsanity.com/paper/2103.00092/) (2021)
- [How Does Data Freshness Affect Real-time Supervised Learning?](https://mlsanity.com/paper/2208.06948/) (2022)
- [The Normalized Cross Density Functional: A Framework to Quantify Statistical Dependence for Random Processes](https://mlsanity.com/paper/2212.04631/) (2022)
- [Minimax Hypothesis Testing for the Bradley-Terry-Luce Model](https://mlsanity.com/paper/2410.08360/) (2024)
- [Learning Cortico-Muscular Dependence through Orthonormal Decomposition of Density Ratios](https://mlsanity.com/paper/2410.14697/) (2024)
- [A Local Geometric Interpretation of Feature Extraction in Deep Feedforward Neural Networks](https://mlsanity.com/paper/2202.04632/) (2022)
- [Group Fairness with Uncertainty in Sensitive Attributes](https://mlsanity.com/paper/2302.08077/) (2023)
- [Dependence Induced Representations](https://mlsanity.com/paper/2411.15328/) (2024)
- [Feature Learning in Image Hierarchies using Functional Maximal Correlation](https://mlsanity.com/paper/2305.20074/) (2023)
- [Kernel Subspace and Feature Extraction](https://mlsanity.com/paper/2301.01410/) (2023)
- [Separable Computation of Information Measures](https://mlsanity.com/paper/2501.15301/) (2025)
- [Multimodal Functional Maximum Correlation for Emotion Recognition](https://mlsanity.com/paper/2512.23076/) (2025)
- [Universal Feature Selection with Noisy Observations and Weak Symmetry Conditions](https://mlsanity.com/paper/2605.09396/) (2026)
- [On the Oracle Complexity of Interpolation-Based Gradient Descent](https://mlsanity.com/paper/2606.19878/) (2026)
- [CB-Agent: Concept-Based In-Context Learning for Sequential Action Selection](https://mlsanity.com/paper/2610.14574/) (2026)

## Abstract (arXiv)

We consider the problem of identifying universal low-dimensional features from high-dimensional data for inference tasks in settings involving learning. For such problems, we introduce natural notions of universality and we show a local equivalence among them. Our analysis is naturally expressed via information geometry, and represents a conceptually and computationally useful analysis. The development reveals the complementary roles of the singular value decomposition, Hirschfeld-Gebelein-R'enyi maximal correlation, the canonical correlation and principle component analyses of Hotelling and Pearson, Tishby's information bottleneck, Wyner's common information, Ky Fan k-norms, and Brieman and Friedman's alternating conditional expectations algorithm. We further illustrate how this framework facilitates understanding and optimizing aspects of learning systems, including multinomial logistic (softmax) regression and the associated neural network architecture, matrix factorization methods for collaborative filtering and other applications, rank-constrained multivariate linear regression, and forms of semi-supervised learning.