The Subaru Deep Field provides the currently deepest K-selected sample of high-z galaxies (K' = 23.5 at 5 sigma). The SDF counts, colors, and size distributions in the near-infrared bands are carefully compared with pure-luminosity-evolution (PLE) as well as CDM-based hierarchical merging (HM) models. The very flat faint-end slope of the SDF K count indicates that the bulk (more than 90%) of cosmic background radiation (CBR) in this band is resolved, even if we take into account every known source of incompleteness. The integrated flux from the counts is only about a third of reported flux of the diffuse CBR in the same band, suggesting that a new distinct source of this missing light may be required. We discovered unusually red objects with colors of (J-K) >~ 3-4, which are even redder than the known population of EROs, and difficult to explain by passively evolving elliptical galaxies. A plausible interpretation, which is the only viable one among those we examined, is that these are dusty starbursts at high-z (z ~ 3), whose number density is comparable with that of present-day ellipticals or spheroidal galaxies, as well as with that of faint submillimeter sources. The photometric redshift distribution obtained by BVRIz'JK' photometries is also compared with the data, and the HM model is found to predict too few high-z objects at K' >~ 22 and z >~ 2; the PLE model with reasonable amount of absorption by dust looks more consistent with the data. This result is apparently in contradiciton with some previous ones for shallower observations, and we discuss the origin of this. These results raise a question for the HM models: how to form massive objects with starbursts at such high redshifts, which presumably evolve into present-day elliptical galaxies or bulges?