Recently, there has been increased need and interest in ”video analysis” which is analysis of video in sequence to determine relatively moving objects like vehicles and different behaviors of people. for eg . this can be used in CCTV network to detect and track abnormal behavior of some people or vehicles. The proposed system, can apply in home and business surveillance system to detect and track moving objects. and also differentiate that, the detected objects are either vehicle or human beings. It is necessary that video surveillance system must detect and track moving object robustly against disturbances birds, trees, environmental changes like different weather conditions etc. so the proposed method is using color background modeling with sensitivity parameter(delta)to remove noises and to detect and track moving objects very easily. Also haar like feature extraction method is used in object recognition. Blob labeling is also used for grouping of moving objects. Then morphological operations like dilation and erosion is also used to remove noises under surveillance. Finally the experiments shows that the proposed method has the robustness against the environmental disturbances and speed which are suitable for the real-time surveillance system.