# Learning Internal Representations (COLT 1995)

arXiv: https://arxiv.org/abs/1911.05781 · HTML: https://mlsanity.com/paper/1911.05781/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2019.

## Built on by (most cited first, 25 of 25)

- [Tip-Adapter: Training-free Adaption of CLIP for Few-shot Classification](https://mlsanity.com/paper/2207.09519/) (2022)
- [Overcoming a Theoretical Limitation of Self-Attention](https://mlsanity.com/paper/2202.12172/) (2022)
- [Interpretable Scientific Discovery with Symbolic Regression: A Review](https://mlsanity.com/paper/2211.10873/) (2022)
- [Sample Efficient Linear Meta-Learning by Alternating Minimization](https://mlsanity.com/paper/2105.08306/) (2021)
- [Multi-Task Imitation Learning for Linear Dynamical Systems](https://mlsanity.com/paper/2212.00186/) (2022)
- [Latent Positional Information is in the Self-Attention Variance of Transformer Language Models Without Positional Embeddings](https://mlsanity.com/paper/2305.13571/) (2023)
- [Improving Context-Based Meta-Reinforcement Learning with Self-Supervised Trajectory Contrastive Learning](https://mlsanity.com/paper/2103.06386/) (2021)
- [Multi-task learning for virtual flow metering](https://mlsanity.com/paper/2103.08713/) (2021)
- [Joint Representation Training in Sequential Tasks with Shared Structure](https://mlsanity.com/paper/2206.12441/) (2022)
- [From Low Intrinsic Dimensionality to Non-Vacuous Generalization Bounds in Deep Multi-Task Learning](https://mlsanity.com/paper/2501.19067/) (2025)
- [IMEX-Reg: Implicit-Explicit Regularization in the Function Space for Continual Learning](https://mlsanity.com/paper/2404.18161/) (2024)
- [Random Forest Autoencoders for Guided Representation Learning](https://mlsanity.com/paper/2502.13257/) (2025)
- [Modern Neural Networks for Small Tabular Datasets: The New Default for Field-Scale Digital Soil Mapping?](https://mlsanity.com/paper/2508.09888/) (2025)
- [PGT-I: Scaling Spatiotemporal GNNs with Memory-Efficient Distributed Training](https://mlsanity.com/paper/2507.11683/) (2025)
- [A Versatile Framework for Analyzing Galaxy Image Data by Implanting Human-in-the-loop on a Large Vision Model](https://mlsanity.com/paper/2405.10890/) (2024)
- [Data-driven analysis of central bank digital currency (CBDC) projects drivers](https://mlsanity.com/paper/2102.11807/) (2021)
- [Learning to Transfer: A Foliated Theory](https://mlsanity.com/paper/2107.10763/) (2021)
- [Newton methods based convolution neural networks using parallel processing](https://mlsanity.com/paper/2112.01401/) (2021)
- [ExpertNet: A Symbiosis of Classification and Clustering](https://mlsanity.com/paper/2201.06344/) (2022)
- [Parameterized Neural Networks for Finance](https://mlsanity.com/paper/2304.08883/) (2023)
- [A Quantitative Approach to Predicting Representational Learning and Performance in Neural Networks](https://mlsanity.com/paper/2307.07575/) (2023)
- [Explainable Time Series Prediction of Tyre Energy in Formula One Race Strategy](https://mlsanity.com/paper/2501.04067/) (2025)
- [Sharper Risk Bound for Multi-Task Learning with Multi-Graph Dependent Data](https://mlsanity.com/paper/2502.18167/) (2025)
- [Federated Learning with Unlabeled Clients: Personalization Can Happen in Low Dimensions](https://mlsanity.com/paper/2505.15579/) (2025)
- [A result relating convex n-widths to covering numbers with some applications to neural networks](https://mlsanity.com/paper/2512.04912/) (2025)

## Abstract (arXiv)

Probably the most important problem in machine learning is the preliminary biasing of a learner's hypothesis space so that it is small enough to ensure good generalisation from reasonable training sets, yet large enough that it contains a good solution to the problem being learnt. In this paper a mechanism for em automatically learning or biasing the learner's hypothesis space is introduced. It works by first learning an appropriate em internal representation for a learning environment and then using that representation to bias the learner's hypothesis space for the learning of future tasks drawn from the same environment. An internal representation must be learnt by sampling from em many similar tasks, not just a single task as occurs in ordinary machine learning. It is proved that the number of examples m em per task required to ensure good generalisation from a representation learner obeys m = O(a+b/n) where n is the number of tasks being learnt and a and b are constants. If the tasks are learnt independently (em i.e. without a common representation) then m=O(a+b). It is argued that for learning environments such as speech and character recognition bgg a and hence representation learning in these environments can potentially yield a drastic reduction in the number of examples required per task. It is also proved that if n = O(b) (with m=O(a+b/n)) then the representation learnt will be good for learning novel tasks from the same environment, and that the number of examples required to generalise well on a novel task will be reduced to O(a) (as opposed to O(a+b) if no representation is used). It is shown that gradient descent can be used to train neural network representations and experiment results are reported providing strong qualitative support for the theoretical results.