To produce a realistic augmentation in Augmented Reality, the correct relative positions of real objects and virtual objects are very important. In this paper, we propose a novel real-time occlusion handling method based on an object tracking approach. Our method is divided into three steps: selection of the occluding object, object tracking and occlusion handling. The user selects the occluding object using an interactive segmentation method. The contour of the selected object is then tracked in the subsequent frames in real-time. In the occlusion handling step, all the pixels on the tracked object are redrawn on the unprocessed augmented image to produce a new synthesized image in which the relative position between the real and virtual object is correct. The proposed method has several advantages. First, it is robust and stable, since it remains effective when the camera is moved through large changes of viewing angles and volumes or when the object and the background have similar colors. Second, it is fast, since the real object can be tracked in real-time. Last, a smoothing technique provides seamless merging between the augmented and virtual object. Several experiments are provided to validate the performance of the proposed method.
References
[1]
Soler, L.; Nicolau, S.; Schmid, J.; Koehl, C.; Marescaux, J.; Pennec, X.; Ayache, N. Virtual reality and augmented reality in digestive surgery. Proceedings of ISMAR, Arlington, VA, USA; 2004; pp. 278–279.
[2]
Thomas, B.; Close, B.; Donoghue, J.; Squires, J. ARQuake: An outdoor/indoor augmented reality first person application. Proceedings of the Fourth International Symposium on Wearable Computers, Atlanta, GA, USA; 2000; pp. 139–146.
[3]
Fjeld, M.; Voegtli, B.M. Augmented chemistry: an interactive educational workbench. Proceedings of International Symposium on Mixed and Augmented Reality, Darmstadt, Germany; 2002; pp. 259–260.
[4]
Vlahakis, V.; Karigiannis, J.; Ioannidis, N. Archeoguide: challenges and solutions of a personalised augmented reality guide for archaeological sites. IEEE Comput. Graph. Appl？2003, 22, 52–60.
[5]
Julier, S.; Baillot, Y.; Lanzagorta, M.; Brown, D.; Rosenblum, L. Bars: battlefield augmented reality system. Proceedings of NATO Symposium on Information Processing Techniques for Military Systems, Istanbul, Turkey; 2000.
[6]
Lee, W.; Park, J. Augmented foam: a tangible augmented reality for product design. Proceedings of ISMAR, Vienna, Austria; 2005; pp. 106–109.
[7]
David, Y.; Efron, U. The image transceiver device: studies of improved physical design. Sensors？2008, 8, 4350–4364.
[8]
Tao, G.; Li, L.J.; Cheng, W. Registration using multiplanar structures for augmented reality systems. J. Comput. Inf. Sci. Eng？2008, 8, 041002-1–041002-6.
[9]
Li, L.J.; Guan, T.; Ren, B.; Yao, X.W.; Wang, C. Registration based on Euclidean reconstruction and natural features tracking for augmented reality systems. Assem. Autom？2008, 28, 340–347.
[10]
Guan, T.; Wang, C. Registration based on scene recognition and natural features tracking techniques for wide-area augmented reality systems. IEEE Trans. Multimedia？2009, 11, 1393–1406.
[11]
Duan, L.Y.; Guan, T.; Yang, B. Registration combining wide and narrow baseline feature tracking techniques for markerless AR systems. Sensors？2009, 9, 10097–10116.
[12]
Fuhrmann, A.; Hesina, G.; Faure, F.; Gervautz, M. Occlusion in collaborative augmented environments. Comput. Graph.-UK？1999, 23, 809–819.
[13]
Ong, K.C.; the, H.C.; Tan, T.S. Resolving occlusion in image sequence made easy. Visual Comput？1998, 14, 153–165.
[14]
Lepetit, V.; Berger, M.O. A semi-automatic method for resolving occlusion in augmented reality. Proceedings of CVPR, Hilton Head Island, SC, USA; 2000; pp. 225–230.
[15]
Berger, M.O. Resolving occlusion in augmented reality: a contour based approach without 3D reconstruction. Proceedings of CVPR, San Juan, Puerto Rico; 1997; pp. 91–96.
[16]
Schmidt, J.; Niemann, H.; Vogt, S. Dense disparity maps in real-time with an application to augmented reality. Proceedings of Sixth IEEE Workshop on Applications of Computer Vision, Orlando, FL, USA; 2002; pp. 225–230.
[17]
Hayashi, K.; Kato, H.; Nishida, S. Occlusion detection of real objects using contour based stereo matching. Proceedings of ICAT, Christchurch, New Zealand; 2005; pp. 180–186.
[18]
Fortin, P.; Hebert, P. Handling occlusions in real-time augmented reality: dealing with movable real and virtual objects. Proceedings of the 3rd Canadian Conference on Computer and Robot Vision, Quebec City, QC, Canada; 2006; p. 54.
[19]
Tian, Y.; Guan, T.; Wang, C.; Li, L.J.; Liu, W. Interactive foreground segmentation method using mean shift and graph cuts. Sens. Rev？2009, 29, 157–162.
[20]
Comaniciu, D.; Meer, P. Mean shift: a robust approach toward feature space analysis. IEEE Trans. Pattern Anal. Mach. Intell？2002, 24, 603–619.
[21]
Cheng, Y.Z. Mean shift, mode seeking, and clustering. IEEE Trans. Pattern Anal. Mach. Intell？1995, 17, 790–799.
[22]
Boykov, Y.Y.; Jolly, M.P. Interactive graph cuts for optimal boundary and region segmentation of objects in N-D images. Proceedings of ICCV, Vancouver, BC, Canada; 2001; pp. 105–112.
[23]
Boykov, Y.; Kolmogorov, V. An experimental comparison of min-cut/max-flow algorithms for energy minimization in computer vision. International Workshop on Energy Minimization Methods in Computer Vision and Pattern Recognition, Sophia Antipolis, France, 2001, 359–374.
[24]
Xu, N.; Bansal, R.; Ahuja, N. Object segmentation using graph cuts based active contours. Proceedings of IEEE International Conference on Computer Vision and Pattern Recognition, Madison, USA; 2003; pp. 46–53.
[25]
Lombaert, H.; Sun, Y.Y.; Grady, L.; Xu, C.Y. A multilevel banded graph cuts method for fast image segmentation. Proceedings of ICCV, Beijing, China; 2005; pp. 259–265.
[26]
Lingua, A.; Marenchino, D.; Nex, F. Performance analysis of the SIFT operator for a Automatic feature extraction and matching in photogrammetric applications. Sensors？2009, 9, 3745–3766.
[27]
Pyramidal implementation of the Lucas Kanade feature tracker. Available online: http://sourceforge.net/projects/opencvlibrary/ (accessed on 2 March 2010).
[28]
ARToolKit. Available online: http://www.hitl.washington.edu/artoolkit/documentation/ (accessed on 2 March 2010).
[29]
OpenCV. Available online: http://www.intel.com/research/mrl/research/opencv (accessed on 2 December 2009).
[30]
Zhang, Z. A flexible new technique for camera calibration. IEEE Trans. Pattern Anal. Mach. Intell？2000, 22, 1330–1334.
