Fast and reliable detection of moving objects is one of the important requirements for many computer vision and video analysis applications. Mean shift based non-parametric background modeling supports more sensitive and robust detection in dynamic outdoor scenes. However it is prohibitive to real-time applications such as video surveillance. This paper aims to deal with the limitation of high computational complexity. Firstly, coarse to fine methods are proposed to avoid raster scanning entire image. Foreground pixels are detected in coarse level to roughly locate the foreground objects in the image, and then fine detection is performed on the corresponding blocks gradually. Secondly， fast mean shift approach is presented according to temporal dependencies. Mean shift iterations are performed starting from incoming data and the modes obtained last time. The experimental results show that the proposed algorithm is effective and efficient in dynamic environment. The proposed algorithm has been applied to move objects detection in our real-time marine video surveillance system.