# Knowledge Matters: Importance of Prior Information for Optimization

arXiv: https://arxiv.org/abs/1301.4083 · HTML: https://mlsanity.com/paper/1301.4083/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Jan 2013.

## Built on by (most cited first, 15 of 15)

- [The pitfalls of next-token prediction](https://mlsanity.com/paper/2403.06963/) (2024)
- [Cross-Modality Knowledge Distillation Network for Monocular 3D Object Detection](https://mlsanity.com/paper/2211.07171/) (2022)
- [Sub-Task Decomposition Enables Learning in Sequence to Sequence Tasks](https://mlsanity.com/paper/2204.02892/) (2022)
- [Contrastive Neural Architecture Search with Neural Architecture Comparators](https://mlsanity.com/paper/2103.05471/) (2021)
- [A Review of Emerging Research Directions in Abstract Visual Reasoning](https://mlsanity.com/paper/2202.10284/) (2022)
- [Imagine the Unseen World: A Benchmark for Systematic Generalization in Visual World Models](https://mlsanity.com/paper/2311.09064/) (2023)
- [Deep Neural Networks Can Learn Generalizable Same-Different Visual Relations](https://mlsanity.com/paper/2310.09612/) (2023)
- [Pose Prior Learner: Unsupervised Categorical Prior Learning for Pose Estimation](https://mlsanity.com/paper/2410.03858/) (2024)
- [Knowledge Distillation with Deep Supervision](https://mlsanity.com/paper/2202.07846/) (2022)
- [Memory Classifiers: Two-stage Classification for Robustness in Machine Learning](https://mlsanity.com/paper/2206.05323/) (2022)
- [Linearly Constrained Weights: Reducing Activation Shift for Faster Training of Neural Networks](https://mlsanity.com/paper/2403.13833/) (2024)
- [Learning richness modulates equality reasoning in neural networks](https://mlsanity.com/paper/2503.09781/) (2025)
- [Auto-Encoder Neural Network Incorporating X-Ray Fluorescence Fundamental Parameters with Machine Learning](https://mlsanity.com/paper/2210.12239/) (2022)
- [A Bayesian Robust Regression Method for Corrupted Data Reconstruction](https://mlsanity.com/paper/2212.12787/) (2022)
- [PARM: Pipeline-Adapted Reward Model](https://mlsanity.com/paper/2604.18327/) (2026)

## Abstract (arXiv)

We explore the effect of introducing prior information into the intermediate level of neural networks for a learning task on which all the state-of-the-art machine learning algorithms tested failed to learn. We motivate our work from the hypothesis that humans learn such intermediate concepts from other individuals via a form of supervision or guidance using a curriculum. The experiments we have conducted provide positive evidence in favor of this hypothesis. In our experiments, a two-tiered MLP architecture is trained on a dataset with 64x64 binary inputs images, each image with three sprites. The final task is to decide whether all the sprites are the same or one of them is different. Sprites are pentomino tetris shapes and they are placed in an image with different locations using scaling and rotation transformations. The first part of the two-tiered MLP is pre-trained with intermediate-level targets being the presence of sprites at each location, while the second part takes the output of the first part as input and predicts the final task's target binary event. The two-tiered MLP architecture, with a few tens of thousand examples, was able to learn the task perfectly, whereas all other algorithms (include unsupervised pre-training, but also traditional algorithms like SVMs, decision trees and boosting) all perform no better than chance. We hypothesize that the optimization difficulty involved when the intermediate pre-training is not performed is due to the em composition of two highly non-linear tasks. Our findings are also consistent with hypotheses on cultural learning inspired by the observations of optimization problems with deep learning, presumably because of effective local minima.