We give a detailed description of the model construction procedures about our new approach to the family structure of the standard model. SM-like chiral fermion spectra, largely "derivable" from the gauge anomaly constraints, are formulated in a $SU(N)\otimes SU(3)\otimes SU(2)\otimes U(1)$ symmetry framework as an extension of the SM symmetry. The $N=4$ case gives naturally three families as a result, with $U(1)_Y$ nontrivially embedded into the $SU(4)_A\otimes U(1)_X$. Such a spectrum has extra vector-like quarks and leptons. We illustrate how an acceptable symmetry breaking pattern can be obtained through a relatively simple scalar sector which gives naturally hierarchical quark mass matrices. Compatibility with various FCNC constraints and some interesting aspects of the possible phenomenological features are discussed, from a non-model specific perspective. The question of incorporating supersymmetry without putting in the Higgses as extra supermultiplet is also addressed.