A complete video surveillance system for automatically tracking shape and position of objects in traffic scenarios is presented. The system, called Auto GMM-SAMT, consists of a detection and a tracking unit. The detection unit is composed of a Gaussian mixture model- (GMM-) based moving foreground detection method followed by a method for determining reliable objects among the detected foreground regions using a projective transformation. Unlike the standard GMM detection the proposed detection method considers spatial and temporal dependencies as well as a limitation of the standard deviation leading to a faster update of the mixture model and to smoother binary masks. The binary masks are transformed in such a way that the object size can be used for a simple but fast classification. The core of the tracking unit, named GMM-SAMT, is a shape adaptive mean shift- (SAMT-) based tracking technique, which uses Gaussian mixture models to adapt the kernel to the object shape. GMM-SAMT returns not only the precise object position but also the current shape of the object. Thus, Auto GMM-SAMT achieves good tracking results even if the object is performing out-of-plane rotations.