Some convergent results for Backtracking Gradient Descent method on Banach spaces

Our main result concerns the following condition: bf Condition C. Let X be a Banach space. A C1 function f:Xrightarrow ℝ satisfies Condition C if whenever xn weakly converges to x and lim nrightarrow∞||nabla f(xn)||=0, then nabla f(x)=0. We assume that there is given a canonical isomorphism between X and its dual X^*, for example when X is a Hilbert space. bf Theorem. Let X be a reflexive, complete Banach space and f:Xrightarrow ℝ be a C2 function which satisfies Condition C. Moreover, we assume that for every bounded set Ssubset X, then sup x∈ S||nabla 2f(x)||<∞. We choose a random point x0∈ X and construct by the Local Backtracking GD procedure (which depends on 3 hyper-parameters α ,β ,δ 0, see later for details) the sequence xn+1=xn-δ (xn)nabla f(xn). Then we have: 1) Every cluster point of xn, in the bf weak topology, is a critical point of f. 2) Either lim nrightarrow∞f(xn)=-∞ or lim nrightarrow∞||xn+1-xn||=0. 3) Here we work with the weak topology. Let C be the set of critical points of f. Assume that C has a bounded component A. Let B be the set of cluster points of xn. If Bcap Anot= emptyset, then Bsubset A and B is connected. 4) Assume that X is separable. Then for generic choices of α ,β ,δ 0 and the initial point x0, if the sequence xn converges - in the bf weak topology, then the limit point cannot be a saddle point.