Hierarchical Models as Marginals of Hierarchical Models

We investigate the representation of hierarchical models in terms of marginals of other hierarchical models with smaller interactions. We focus on binary variables and marginals of pairwise interaction models whose hidden variables are conditionally independent given the visible variables. In this case the problem is equivalent to the representation of linear subspaces of polynomials by feedforward neural networks with soft-plus computational units. We show that any binary hierarchical model with $M$ pure higher order interactions can be expressed as the marginal of a pairwise interaction model with $\sim \tfrac12 M$ hidden binary variables.