Extending Universal Approximation Guarantees: A Theoretical Justification for the Continuity of Real-World Learning Tasks

Universal Approximation Theorems establish the density of various classes of neural network function approximators in C(K, ℝm), where K subset ℝn is compact. In this paper, we aim to extend these guarantees by establishing conditions on learning tasks that guarantee their continuity. We consider learning tasks given by conditional expectations x mapsto Eleft[Y mid X = xright], where the learning target Y = f circ L is a potentially pathological transformation of some underlying data-generating process L. Under a factorization L = T circ W for the data-generating process where T is thought of as a deterministic map acting on some random input W, we establish conditions (that might be easily verified using knowledge of T alone) that guarantee the continuity of practically textitany derived learning task x mapsto Eleft[f circ L mid X = xright]. We motivate the realism of our conditions using the example of randomized stable matching, thus providing a theoretical justification for the continuity of real-world learning tasks.