mlsanityData to 14 Oct 2026

NeuronSeek: On Stability and Expressivity of Task-driven Neurons

arXiv 2506.15715 ↗First posted 1 Jun 2025cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Drawing inspiration from our human brain that designs different neurons for different tasks, recent advances in deep learning have explored modifying a network's neurons to develop so-called task-driven neurons. Prototyping task-driven neurons (referred to as NeuronSeek) employs symbolic regression (SR) to discover the optimal neuron formulation and construct a network from these optimized neurons. Along this direction, this work replaces symbolic regression with tensor decomposition (TD) to discover optimal neuronal formulations, offering enhanced stability and faster convergence. Furthermore, we establish theoretical guarantees that modifying the aggregation functions with common activation functions can empower a network with a fixed number of parameters to approximate any continuous function with an arbitrarily small error, providing a rigorous mathematical foundation for the NeuronSeek framework. Extensive empirical evaluations demonstrate that our NeuronSeek-TD framework not only achieves superior stability, but also is competitive relative to the state-of-the-art models across diverse benchmarks. The code is available at https://github.com/HanyuPei22/NeuronSeek.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 45 of its 82 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

45 of 82 references.

References · 45

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2016Densely Connected Convolutional Networkscited by 6,720
  4. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,166
  5. 2017Squeeze-and-Excitation Networkscited by 4,789
  6. 2014Going Deeper with Convolutionscited by 4,241
  7. 2020Fourier Features Let Networks Learn High Frequency Functions in Low Dimensional Domainscited by 2,200
  8. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  9. 2019TabNet: Attentive Interpretable Tabular Learningcited by 648
  10. 2018Machine Learning for Combinatorial Optimization: a Methodological Tour d'Horizoncited by 623
  11. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  12. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
33 more
  1. 2019Deep Learning for Symbolic Mathematicscited by 270
  2. 2021A Survey on Deep Semi-supervised Learningcited by 251
  3. 2022End-to-end symbolic regression with transformerscited by 203
  4. 2020The Next Decade in AI: Four Steps Towards Robust Artificial Intelligencecited by 195
  5. 2024U-KAN Makes Strong Backbone for Medical Image Segmentation and Generationcited by 179
  6. 2019Deep Learning for Spatio-Temporal Data Mining: A Surveycited by 172
  7. 2018Learning Equations for Extrapolation and Controlcited by 156
  8. 2023Neural Architecture Search: Insights from 1000 Paperscited by 139
  9. 2016Extrapolation and learning equationscited by 110
  10. 2021SymbolicGPT: A Generative Transformer Model for Symbolic Regressioncited by 105
  11. 2022Interpretable Scientific Discovery with Symbolic Regression: A Reviewcited by 105
  12. 2019Integration of Neural Network-Based Symbolic Regression in Deep Learning for Scientific Discoverycited by 93
  13. 2022Toward Next-Generation Artificial Intelligence: Catalyzing the NeuroAI Revolutioncited by 78
  14. 2024From PINNs to PIKANs: Recent Advances in Physics-Informed Machine Learningcited by 73
  15. 2020From Statistical Relational to Neuro-Symbolic Artificial Intelligencecited by 63
  16. 2020Deep Polynomial Neural Networkscited by 63
  17. 2020Neural Network Approximation: Three Hidden Layers Are Enoughcited by 63
  18. 2021DANets: Deep Abstract Networks for Tabular Data Classification and Regressioncited by 50
  19. 2021Quadratic Residual Networks: A New Class of Neural Networks for Solving Forward and Inverse Problems in Physics Involving PDEscited by 43
  20. 2021Deep Network Approximation: Achieving Arbitrary Accuracy with Fixed Number of Neuronscited by 41
  21. 2017Non-linear Convolution Filters for CNN-based Learningcited by 38
  22. 2021Elementary superexpressive activationscited by 36
  23. 2022Exhaustive Symbolic Regressioncited by 29
  24. 2021Augmenting Deep Classifiers with Polynomial Neural Networkscited by 21
  25. 2022QuadraLib: A Performant Quadratic Neural Network Library for Architecture Optimization and Design Explorationcited by 21
  26. 2021On Expressivity and Trainability of Quadratic Networkscited by 19
  27. 2023A Framework Based on Symbolic Regression Coupled with eXtended Physics-Informed Neural Networks for Gray-Box Learning of Equations of Motion from Datacited by 18
  28. 2017A New Type of Neurons for Machine Learningcited by 16
  29. 2023Regularization of polynomial networks for image recognitioncited by 13
  30. 2019Randomized Tensor Ring Decomposition and Its Application to Large-scale Data Reconstructioncited by 11
  31. 2023Towards NeuroAI: Introducing Neuronal Diversity into Artificial Neural Networkscited by 6
  32. 2023On Enhancing Expressive Power via Compositions of Single Fixed-Size ReLU Networkcited by 6
  33. 2024No One-Size-Fits-All Neurons: Task-based Neurons for Artificial Neural Networkscited by 3

Built on by

Not cited yet.