# Objective-Specific Privileged Bases via Full-Prefix Matryoshka Learning

arXiv: https://arxiv.org/abs/2605.09160 · HTML: https://mlsanity.com/paper/2605.09160/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 9 of its 14 references that are in the map.
- First posted on arXiv 9 May 2026.

## Builds on (most cited first, 9 of 9)

- [Matryoshka Representation Learning](https://mlsanity.com/paper/2205.13147/) (2022)
- [Learning Ordered Representations with Nested Dropout](https://mlsanity.com/paper/1402.0915/) (2014)
- [Matryoshka Multimodal Models](https://mlsanity.com/paper/2405.17430/) (2024)
- [Matryoshka Query Transformer for Large Vision-Language Models](https://mlsanity.com/paper/2405.19315/) (2024)
- [Loss Landscapes of Regularized Linear Autoencoders](https://mlsanity.com/paper/1901.08168/) (2019)
- [MetaEmbed: Scaling Multimodal Retrieval at Test-Time with Flexible Late Interaction](https://mlsanity.com/paper/2509.18095/) (2025)
- [Regularized linear autoencoders recover the principal components, eventually](https://mlsanity.com/paper/2007.06731/) (2020)
- [Matryoshka Model Learning for Improved Elastic Student Models](https://mlsanity.com/paper/2505.23337/) (2025)
- [PCAE: Learning Ordered Representations in Latent Space for Intrinsic Dimension Estimation via Principal Component Autoencoder](https://mlsanity.com/paper/2601.19179/) (2026)

## Abstract (arXiv)

Learned representations are often invariant to rotational transformations, leaving individual dimensions non-identifiable and interchangeable. We study how Matryoshka Representation Learning (MRL) induces a task-aligned privileged basis distinct from variance-based or regularizer-induced orderings. In the linear setting, we prove that full-prefix MRL recovers the ordered principal directions, and can be computed efficiently using shared statistics. Empirically, we demonstrate that MRL yields consistent per-dimension structure aligned with task signal, where coordinate magnitude reflects informativeness.