The enigmatic pseudogap phase in underdoped cuprate high T_c superconductors has long been recognized as a central puzzle of the T_c problem. Recent data show that the pseudogap is likely a distinct phase, characterized by a medium range and quasi-static charge ordering. However, the origin of the ordering wavevector and the mechanism of the charge order is unknown. At the same time, earlier data show that precursive superconducting fluctuations are also associated with this phase. We propose that the pseudogap phase is a novel pairing state where electrons on the same side of the Fermi surface are paired, in strong contrast with conventional BCS theory which pair electrons on opposite sides of the Fermi surface. In this state the Cooper pair carries a net momentum and belong to a general class called pair density wave (PDW). The microscopic pairing mechanism comes from a gauge theory formulation of the resonating valence bond (RVB) picture, where electrons traveling in the same direction feel an attractive force in analogy with Ampere's effects in electromagnetism. We call this Amperean pairing. Charge order automatically appears as a subsidiary order parameter. Our theory gives a prediction of the ordering wavevector which is in good agreement with experiment. Furthermore, the quasiparticle spectrum from our model explains many of the unusual features reported in photoemission experiments. The Fermi arc and the unusual way the tip of the arc terminates also come out naturally. We also discuss how the onset of the Kerr effect in this state can be accommodated. Finally, we propose an experiment which can directly test the notion of Amperean pairing.