Dimension-Independent Kernel ε-Covers

We introduce the notion of an varepsilon-cover for a kernel range space. A kernel range space concerns a set of points X subset ℝd and the space of all queries by a fixed kernel (e.g., a Gaussian kernel K(p,·) = exp(-|p-·|2), where p ∈ ℝd). For a point set X of size n, a query returns a vector of values Rp ∈ ℝn, where the ith coordinate (Rp)i = K(p,xi) for xi ∈ X. An varepsilon-cover is a subset of points Q subset ℝd so for any p ∈ ℝd that frac1n |Rp - Rq|1≤ varepsilon for some q ∈ Q. This is a smooth analog of Haussler's notion of varepsilon-covers for combinatorial range spaces (e.g., defined by subsets of points within a ball query) where the resulting vectors Rp are in 0,1n instead of [0,1]n. The kernel versions of these range spaces show up in data analysis tasks where the coordinates may be uncertain or imprecise, and hence one wishes to add some flexibility in the notion of inside and outside of a query range. Our main result is that, unlike combinatorial range spaces, the size of kernel varepsilon-covers is independent of the input size n and dimension d. We obtain a bound of 2tilde O(1/varepsilon2), where tildeO(f(1/varepsilon)) hides log factors in (1/varepsilon) that can depend on the kernel. This implies that by relaxing the notion of boundaries in range queries, eventually the curse of dimensionality disappears, and may help explain the success of machine learning in very high-dimensions. We also complement this result with a lower bound of almost (1/varepsilon)Omega(1/varepsilon), showing the exponential dependence on 1/varepsilon is necessary.