# Limitations of Learning Tanh Neural Networks with Finite Precision

arXiv: https://arxiv.org/abs/2606.11104 · HTML: https://mlsanity.com/paper/2606.11104/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 27 references that are in the map.
- First posted on arXiv 9 Jun 2026.

## Builds on (most cited first, 14 of 14)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Fundamentals of Recurrent Neural Network (RNN) and Long Short-Term Memory (LSTM) Network](https://mlsanity.com/paper/1808.03314/) (2018)
- [Deep Neural Network Approximation Theory](https://mlsanity.com/paper/1901.02220/) (2019)
- [Optimal Approximation with Sparsely Connected Deep Neural Networks](https://mlsanity.com/paper/1705.01714/) (2017)
- [On the approximation of functions by tanh neural networks](https://mlsanity.com/paper/2104.08938/) (2021)
- [A proof that artificial neural networks overcome the curse of dimensionality in the numerical approximation of Black-Scholes partial differential equations](https://mlsanity.com/paper/1809.02362/) (2018)
- [Analysis of the Generalization Error: Empirical Risk Minimization over Deep Artificial Neural Networks Overcomes the Curse of Dimensionality in the Numerical Approximation of Black-Scholes Partial Differential Equations](https://mlsanity.com/paper/1809.03062/) (2018)
- [Proof of the Theory-to-Practice Gap in Deep Learning via Sampling Complexity bounds for Neural Network Approximation Spaces](https://mlsanity.com/paper/2104.02746/) (2021)
- [Robust Implicit Networks via Non-Euclidean Contractions](https://mlsanity.com/paper/2106.03194/) (2021)
- [Learning ReLU networks to high uniform accuracy is intractable](https://mlsanity.com/paper/2205.13531/) (2022)
- [Non-Euclidean Contraction Analysis of Continuous-Time Neural Networks](https://mlsanity.com/paper/2110.08298/) (2021)
- [Sobolev-type embeddings for neural network approximation spaces](https://mlsanity.com/paper/2110.15304/) (2021)

## Built on by (most cited first, 1 of 1)

- [The Information-Theoretic Benefit of Shared Representations under Orthogonality Constraints](https://mlsanity.com/paper/2606.16028/) (2026)

## Abstract (arXiv)

We investigate limitations of learning tanh neural networks from point evaluations under finite-precision computations and Lp accuracy guarantees, building on Berner, Grohs, and Voigtl"ander (2023). Our approach is based on a novel construction of sharply localized bump functions via iterated tanh activations. Using this mechanism, we show that, in a finite-precision setting, no adaptive randomized algorithm based on m samples can achieve a convergence rate higher than the Monte Carlo rate O(m-1/p) in the Lp norm, unless the sampling budget grows exponentially with the size of the network parameters and architecture. The results reveal fundamental limitations imposed by finite precision on the learnability of classes containing localized bump functions, extending previous results for ReLU networks to the tanh setting.