How to Open Kernel?

Gaussian Process Models: How can a kernel function act both as a covariance function in a space, and as an inner product in another space?

• Kernel functions have been an interest of mine for more than a year now, yet I still can not grasp some of the subtleties that they imply. My first introduction with kernel functions defined them as inner products in some mapped space [math]k(x, y) = \langle \phi(x), \phi(y) \rangle [/math], and I believe that most people in ML have received the same first-hand introduction. However, exactly the same kernels are also often used to indicate/define the covariance between two random variables / vectors, i.e. [math]k(X, Y) = Cov(X, Y)[/math]. What is the relationship between the two uses? I know there is a very deep connection hidden somewhere, but I have never understood which. I am pretty sure that the kernel acts as inner product in a certain space, and as a covariance in another, but I really do not know the relationship between the two spaces. Please help me / point me to an adequate source of knowledge!

• Answer:

  The missing link is the orthogonal representation of the process.  If you can write [math]f(x) = \sum_{i=1}^\infty Z_i h_i(x)[/math] where [math]Z_i[/math] are independent random variables with mean 0 and variance 1, and where [math]h_i[/math] are fixed basis functions, then defining the vector-valued function [math]\phi(x) = (h_1(x),h_2(x),...,)[/math], we have [math]k(x,y)=Cov(f(x),f(y)) [/math] [math]= \sum_{i=1}^\infty Cov(Z_i h_i(x),Z_i h_i(y))[/math] [math] =\sum_{i=1}^\infty h_i(x)h_i(y) = \langle \phi(x),\phi(y)\rangle[/math]