Contractivity of neural ODEs: an eigenvalue optimization problem

We propose a novel methodology to solve a key eigenvalue optimization problem which arises in the contractivity analysis of neural ODEs. When looking at contractivity properties of a one layer weight-tied neural ODE dotu(t)=σ(Au(t)+b) (with u,b ∈ mathbb Rn, A is a given n × n matrix, σ : mathbb R → mathbb R denotes an activation function and for a vector z ∈ mathbb Rn, σ(z) ∈ mathbb Rn has to be interpreted entry-wise), we are led to study the logarithmic norm of a set of products of type D A, where D is a diagonal matrix such that diag(D) ∈ σ'(mathbb Rn). Specifically, given a real number c (usually c=0), the problem consists in finding the largest positive interval Isubseteq mathbb [0,∞) such that the logarithmic norm μ(DA) ≤ c for all diagonal matrices D with Dii∈ I. We propose a two-level nested methodology: an inner level where, for a given I, we compute an optimizer D^star(I) by a gradient system approach, and an outer level where we tune I so that the value c is reached by μ(D^star(I)A). We extend the proposed two-level approach to the general multilayer, and possibly time-dependent, case dotu(t) = σ( Ak(t) ldots σ ( A1(t) u(t) + b1(t) ) ldots + bk(t) ) and we propose several numerical examples to illustrate its behaviour, including its stabilizing performance on a one-layer neural ODE applied to the classification of the MNIST handwritten digits dataset.