Deep Learning: Why is the energy function of the Restricted Boltzmann Machine defined the way it is defined?

This is a good question since the original papers on the Boltzmann Machine (BM) by Sejnowski et al. never showed the mathematical relation between the Boltzmann distribution and the corresponding the activation function. The short answer is that the activation functions can be derived simply by Bayesian analysis. For a BM,  P(x) = exp(-E(x))/(sum_(x’) exp(E(x’) ) )    (1) where:      x = state of the BM        P(x) = probability of x generated by the BM                  (when running at equilibrium)      E(x) = -sum_(ij) x_i x_j  (ignoring biases) = the energy function If you assume the x units are binary and derive P(x_i| x_j: j!= i) (‘!=‘ means ‘not equal’) from equation 1 you derive the sigmoidal stochastic activation function:  P(x_i=1 | x_j: j!= i) = siqmoid( sum( w_ij * x_j: j!= i ) ) )     (2) i.e, the standard activation function of a BM unit  (ignoring biases),  by using Bayes rule with eq. 1 to obtain eq. 2.  A good paper on generalizing the BM, or really, on generalizing Restricted Boltzmann Machines, is by Hinton, Rosen-Zvi, and Welling) called "Exponential Family Harmoniums with an Application to Information Retrieval."  The different forms of RBM’s derived in that paper can be reached by using Bayes rule as above. From eqs. 1  and 2 it's not hard to show that the transition dynamics of a BM obey “detailed balance” where the probability of being on the transition from x to x’ is the same as being on the transition in reverse, i.e., P(x)P(x->x’) = P(x')P(x’->x), where P(x) refers to the probability of being in state x for the equilibrium distribution of the BM. This means that the BM has no information in it with respect to the sequence of the states - at least in equilibrium (it asserts acausality). From a practical point of view it might be easier to use the detailed balance equation to derive, e.g., the activation functions than the method described in the previous paragraph. It is also interesting to consider what kind of distribution will be generated from a fully-connected network like the BM but with with some arbitrarily concocted unit activation function(s). Generally such a distribution will not be in detailed balance, and in fact there may not even be an equilibrium distribution. I am wondering if the author of this question has a mindset somewhat like mine, since this question loomed large in my mind when I first started studying this. Being a mechanical engineer and more apt to think about the "visceral" operation of the BM, it amazed me that its behavior can be reduced to a very simple equation. But at least now I see how it works more viscerally than I used to. The key is that the equilibrium distribution of the BM is implied by the summation over all states in the partition function (the denominator of eq. 1), i.e., the latter is performing an expectation. Perhaps the more interesting  part is how the summation over all the states reduces to a summation over all local states when finding the activation function (eq. 2).

All our efforts in machine learning are devoted to designing discriminators of some kind. That is, we want to design systems which discriminate good inputs from the bad ones. We also want to make sure that the system learns to discriminate between the inputs in a predictable way. One such system is the Energy based model. In energy based model, we tend to create an analogy with the thermodynamic systems. Learning is done by minimizing the system energy for desirable inputs and not minimizing it for the undesirable inputs. Initial effort for creating the energy based models tried to mimic the thermodynamic systems. As always, learning from nature is the best way. You can define your own energy function (or whatever you call it), as far as you can define an efficient learning procedure for it, you are doing it right.

Your question suggests 2 different ways to modify the energy function: (1) different types of units and (2) more types of connections. For the first, you can use whatever distributions which can be described in exponential family and then combine them multiplicatively. The combined distribution will also belongs to exponential family. Then you can decompose the log-likelihood term into data-dependent and data-independent expectations. For the second, you also can add connections among units, but now the model is "unrestricted" and more complicated (e.g., conditional RBM and Boltzmann machine). You will have to resort an approximation way to estimate the data-dependent. The computational cost also increases.

I think though RBM has its roots in Hopfield nets and Ising models, it is also motivated by Boltzmann distribution. If we think about RBM in the context of Boltzmann distribution, finding the energy function is just solving a functional equation in which the current energy function is the simplest solution.