The thermopower and conductance of atomic-size metallic contacts have been simultaneously measured using a mechanically controllable break junction. For contacts approaching atomic dimensions, abrupt steps in the thermopower are observed which coincide with jumps in the conductance. The measured thermopower for a large number of atomic-size contacts is randomly distributed around the value for large contacts and can be either positive or negative in sign. However, it is suppressed at the quantum value of the conductance G_0 = 2e^2/h. We derive an expression that describes these results in terms of quantum interference of electrons backscattered in the banks.