We present VLBA observations at 15 GHz of ten GHz Peaked Spectrum (GPS) radio sources. The cores are often difficult or impossible to locate. When likely cores are found, they account for a small fraction of the flux density in GPS galaxies - around or below 2%, while in GPS quasars they can account for more than 20% of the total flux density. We detect low polarization in the GPS sources -- i.e., typically less than a few percent and often less than one percent. This establishes that low polarization in the parsec scale structure is an important defining characteristic of the GPS sources. The dichotomy in the radio morphology versus optical identification, i.e., galaxies are symmetric and quasars are not, is basically confirmed from these new data, which also indicate that the radio emission from GPS quasars is dominated by a jet, with often a weak or hidden core, suggesting they are at moderate angles to the line of sight, and so are only moderately beamed.