Layer-wise Derivative Controlled Networks Achieve Competitive Accuracy and Gradient Stability Across Data Regimes
Not on the map. Why?
Abstract (arXiv)
Derivative-controlled networks based on ChainzRule (CR) combine cubic polynomial layers with a lightweight forward-mode per-layer Jacobian penalty (DREG). In this second paper of a multi-part series, we evaluate the generalization properties of CR across data regimes. We ablate the shape of the DREG coefficient schedule, demonstrating that the optimal annealing range depends on representation noise. On the Pima Diabetes dataset, CR achieves strong low-data performance and maintains a consistent accuracy advantage over baselines from 5% to 100% training data, supported by exceptionally stable gradient tail ratios (~1.01--1.02 vs. 1.07--1.09 for ReLU networks). Extensions to SST-5 show competitive or superior results in both frozen-embedding and BERT fine-tuned regimes, including outperforming prior BERT baselines despite substantially less training data. These results are statistically significant: CR achieves superior accuracy over the strongest published baselines we could identify on both datasets (p < 0.05). These results establish that layer-wise derivative control induces a structural inductive bias toward low-frequency, stable representations that generalizes robustly across tabular and NLP domains, data volumes, and representation qualities. The gradient tail ratio serves as a reliable, label-free diagnostic of generalization capability.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 7 of its 19 references that are in the map. First posted on arXiv 6 Jun 2026.
Builds on
7 of 19 references.
References · 7
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
- 2024KAN: Kolmogorov-Arnold Networkscited by 634
- 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
- 2017Parseval Networks: Improving Robustness to Adversarial Examplescited by 288
- 2017Sobolev Training for Neural Networkscited by 136
- 2019Fine-grained Sentiment Classification using BERTcited by 38
Built on by
Not cited yet.