mlsanityData to 7 Oct 2026

An initial alignment between neural network and target is needed for gradient descent to learn

arXiv 2202.12846 ↗First posted 25 Feb 2022cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

This paper introduces the notion of ``Initial Alignment'' (INAL) between a neural network at initialization and a target function. It is proved that if a network and a Boolean target function do not have a noticeable INAL, then noisy gradient descent on a fully connected network with normalized i.i.d. initialization will not learn in polynomial time. Thus a certain amount of knowledge about the target (measured by the INAL) is needed in the architecture design. This also provides an answer to an open problem posed in [AS20]. The results are based on deriving lower-bounds for descent algorithms on symmetric neural networks without explicit knowledge of the target function beyond its INAL.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 7 of its 17 references that are in the map. First posted on arXiv 25 Feb 2022.

Builds on

7 of 17 references.

References · 7

  1. 2018An Intriguing Failing of Convolutional Neural Networks and the CoordConv Solutioncited by 302
  2. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  3. 2021On the Power of Differentiable Learning versus PAC and SQ Learningcited by 27
  4. 2019Is Deeper Better only when Shallow is Good?cited by 24
  5. 2020Poly-time universality and limitations of deep learningcited by 23
  6. 2020Computational Separation Between Convolutional and Fully-Connected Networkscited by 19
  7. 2021A cautionary tale on fitting decision trees to data from additive models: generalization lower boundscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 5; 2024: 1; 2025: 1; 2026: 1.

12 papers.

  1. 2022Hidden Progress in Deep Learning: SGD Learns Parities Near the Computational Limitcited by 184
  2. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  3. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  4. 2023Generalization on the Unseen, Logic Reasoning and Degree Curriculumcited by 73
  5. 2024How Far Can Transformers Reason? The Globality Barrier and Inductive Scratchpadcited by 35
  6. 2023When can transformers reason with abstract symbols?cited by 23
  7. 2023Provable Advantage of Curriculum Learning on Parity Targets with Mixed Inputscited by 22
  8. 2023A Mathematical Model for Curriculum Learning for Paritiescited by 19
  9. 2022Learning to Reason with Neural Networks: Generalization, Unseen Data and Boolean Measurescited by 17
  10. 2022On the non-universality of deep learning: quantifying the cost of symmetrycited by 17
  11. 2025How Does Preconditioning Guide Feature Learning in Deep Neural Networks?cited by 0
  12. 2026Pattern Formation in Transformerscited by 0