Towards universal neural nets: Gibbs machines and ACE

We study a class of neural nets - \emph{Gibbs machines} - which are a type of variational auto-encoders, designed for gradual learning. They offer an universal platform for incrementally adding newly learned features, including physical symmetries, and are directly connected to information geometry and thermodynamics. Combining them with classifiers, gives rise to a brand of universal generative neural nets - stochastic auto-classifier-encoders (ACE). ACE have state-of-the-art performance in their class, both for classification and density estimation for the MNIST data set.