mlsanityData to 7 Oct 2026

Information Theoretic Perspective on Representation Learning

arXiv 2601.11334 ↗First posted 16 Jan 2026cs.IT cs.LG math.ITcited by 0

Not on the map. Why?

Abstract (arXiv)

An information-theoretic framework is introduced to analyze last-layer embedding, focusing on learned representations for regression tasks. We define representation-rate and derive limits on the reliability with which input-output information can be represented as is inherently determined by the input-source entropy. We further define representation capacity in a perturbed setting, and representation rate-distortion for a compressed output. We derive the achievable capacity, the achievable representation-rate, and their converse. Finally, we combine the results in a unified setting.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 21 of its 49 references that are in the map. First posted on arXiv 16 Jan 2026.

Builds on

21 of 49 references.

References · 21

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  4. 2020wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representationscited by 3,853
  5. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  6. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  7. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  8. 2021Decoupled Contrastive Learningcited by 141
  9. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  10. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  11. 2020Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Taskscited by 47
  12. 2025Understanding and Mitigating Numerical Sources of Nondeterminism in LLM Inferencecited by 47
9 more
  1. 2023Reverse Engineering Self-Supervised Learningcited by 33
  2. 2025Language Models are Injective and Hence Invertiblecited by 31
  3. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  4. 2019The capacity of feedforward neural networkscited by 14
  5. 2020Memory capacity of neural networks with threshold and ReLU activationscited by 13
  6. 2023Fully Convolutional Slice-to-Volume Reconstruction for Single-Stack MRIcited by 10
  7. 2025Self-Supervised Contrastive Learning is Approximately Supervised Contrastive Learningcited by 5
  8. 2023Domain-Aware Few-Shot Learning for Optical Coherence Tomography Noise Reductioncited by 3
  9. 2024One-Shot Image Restorationcited by 3

Built on by

Not cited yet.