mlsanityData to 14 Oct 2026

Limitations of Learning Tanh Neural Networks with Finite Precision

arXiv 2606.11104 ↗First posted 9 Jun 2026cs.LG math.CA stat.MLcited by 1

Not on the map. Why?

Abstract (arXiv)

We investigate limitations of learning tanh neural networks from point evaluations under finite-precision computations and Lp accuracy guarantees, building on Berner, Grohs, and Voigtl"ander (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated tanh activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on m samples can achieve a convergence rate higher than the Monte Carlo rate O(m-1/p) in the Lp norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the tanh setting.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 14 of its 27 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

14 of 27 references.

References · 14

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  3. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,304
  4. 2018Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Networkcited by 420
  5. 2019Deep Neural Network Approximation Theorycited by 93
  6. 2017Optimal Approximation with Sparsely Connected Deep Neural Networkscited by 91
  7. 2021On the approximation of functions by tanh neural networkscited by 81
  8. 2018A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equationscited by 55
  9. 2018Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equationscited by 45
  10. 2021Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spacescited by 30
  11. 2021Robust Implicit Networks via Non-Euclidean Contractionscited by 20
  12. 2022Learning ReLU networks to high uniform accuracy is intractablecited by 5
2 more
  1. 2021Non-Euclidean Contraction Analysis of Continuous-Time Neural Networkscited by 2
  2. 2021Sobolev-type embeddings for neural network approximation spacescited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraintscited by 0