# A Universal Approximation Result for Difference of log-sum-exp Neural Networks

arXiv: https://arxiv.org/abs/1905.08503 · HTML: https://mlsanity.com/paper/1905.08503/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 May 2019.

## Built on by (most cited first, 9 of 9)

- [Min-Max-Plus Neural Networks](https://mlsanity.com/paper/2102.06358/) (2021)
- [Partial Identification of Causal Effects Using Proxy Variables](https://mlsanity.com/paper/2304.04374/) (2023)
- [A Margin-Maximizing Fine-Grained Ensemble Method](https://mlsanity.com/paper/2409.12849/) (2024)
- [Parameterized Convex Universal Approximators for Decision-Making Problems](https://mlsanity.com/paper/2201.06298/) (2022)
- [Achieving More with Less: A Tensor-Optimization-Powered Ensemble Method](https://mlsanity.com/paper/2408.02936/) (2024)
- [Revisiting Tropical Polynomial Division: Theory, Algorithms and Application to Neural Networks](https://mlsanity.com/paper/2306.15157/) (2023)
- [SPINE: Soft Piecewise Interpretable Neural Equations](https://mlsanity.com/paper/2111.10622/) (2021)
- [A Framework for Analyzing Cross-correlators using Price's Theorem and Piecewise-Linear Decomposition](https://mlsanity.com/paper/2304.09242/) (2023)
- [Parameterized Convex Minorant for Objective Function Approximation in Amortized Optimization](https://mlsanity.com/paper/2310.02519/) (2023)

## Abstract (arXiv)

We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node (LSE networks) is a smooth universal approximator of continuous functions over convex, compact sets. By using a logarithmic transform, this class of networks maps to a family of subtraction-free ratios of generalized posynomials, which we also show to be universal approximators of positive functions over log-convex, compact subsets of the positive orthant. The main advantage of Difference-LSE networks with respect to classical feedforward neural networks is that, after a standard training phase, they provide surrogate models for design that possess a specific difference-of-convex-functions form, which makes them optimizable via relatively efficient numerical methods. In particular, by adapting an existing difference-of-convex algorithm to these models, we obtain an algorithm for performing effective optimization-based design. We illustrate the proposed approach by applying it to data-driven design of a diet for a patient with type-2 diabetes.