Differential equations, splines and Gaussian processes

We explore the connections between Green's functions for certain differential equations, covariance functions for Gaussian processes, and the smoothing splines problem. Conventionally, the smoothing spline problem is considered in a setting of reproducing kernel Hilbert spaces, but here we present a more direct approach. With this approach, some choices that are implicit in the reproducing kernel Hilbert space setting stand out, one example being choice of boundary conditions and more elaborate shape restrictions. The paper first explores the Laplace operator and the Poisson equation and studies the corresponding Green's functions under various boundary conditions and constraints. Explicit functional forms are derived in a range of examples. These examples include several novel forms of the Green's function that, to the author's knowledge, have not previously been presented. Next we present a smoothing spline problem where we penalize the integrated squared derivative of the function to be estimated. We then show how the solution can be explicitly computed using the Green's function for the Laplace operator. In the last part of the paper, we explore the connection between Gaussian processes and differential equations, and show how the Laplace operator is related to Brownian processes and how processes that arise due to boundary conditions and shape constraints can be viewed as conditional Gaussian processes. The presented connection between Green's functions for the Laplace operator and covariance functions for Brownian processes allows us to introduce several new novel Brownian processes with specific behaviors. Finally, we consider the connection between Gaussian process priors and smoothing splines.