Effect of Spherical Parameter Space on Number of Non-Forgetting Tasks and Capacity of Continual Learning

We propose a continuous learning formulation for T non-forgetting tasks with the spherical parameter space mathbbSd-1(√d) in asymptotic regime drightarrow ∞. Each task i has an optimum bmThetai, minimizing its corresponding loss function Ri(bmw) on sphere. We reduce each task i to an acceptable loss region modeled by a geodesic ball B_bmThetai(ri(d)) on mathbbSd-1(√d) centered at bmThetai with radius ri(d). To model task dependencies (similarities), task optima bmThetai are distributed by a Neyman-Scott clustered point process Phi. Cluster centers bmYjj=1Np form a homogeneous Poisson point process on sphere, while intra-cluster task optima follow a von Mises-Fisher distribution with mean bmYj and concentration kappa. Each task t∈ 1, ..., T learning algorithm is considered to be a channel Qt(dbmwt | bmwt-1, Kt-1(bmθ1:t-1), bmθt), where bmWt is the learned parameter of task t and Kt-1(bmTheta1:t-1)=bigcapi=1t-1B_bmThetai(ri(d)). Since in continual learning, the learner does not need to reach bmThetat exactly to successfully learn task t, to avoid forgetting as well as reliable learning, it only needs for the learned parameter vector bmWt to enter B_bmThetat(rt(d)) cap Kt-1(bmTheta1:t-1). Using this model, we derive an upper bound on T ensuring mathbbP[KT(bmTheta1:T) neq emptyset] rightarrow 1 as drightarrow ∞ and kappa/d rightarrow α, where α>0 is a constant. Finally, for T tasks sampled uniformly from Phi, we derive an upper bound for total information theoretic capacity under the non-forgetting constraint for each task.