State-of-the-art object recognition and pose estimation systems often utilize point feature algorithms, which in turn usually require the computing power of conventional PC hardware. In this paper, we describe two embedded systems for object detection and pose estimation using sophisticated point features. The feature detection step of the “Speeded-up Robust Features (SURF)” algorithm is accelerated by a special IP core. The first system performs object detection and is completely implemented in a single medium-size Virtex-5 FPGA. The second system is an augmented reality platform, which consists of an ARM-based microcontroller and intelligent FPGA-based cameras which support the main system. 1. Introduction Computer vision (CV) and augmented reality (AR) are growing areas of research with many applications. For example, automotive industry makes use of passive optical sensors in the field of offboard traffic observation and management [1–3]. Onboard systems often utilize CV techniques for driver assistance and traffic sign detection [4, 5]. Turning CV to account, AR enhances real environments by virtual elements and allows manifold applications such as guided order picking or maintenance tasks [6, 7]. Optical object detection and pose estimation are very challenging tasks since they have to deal with problems such as different views of an object, various light conditions, surface reflections, and noise caused by image sensors. Presently available algorithms such as SIFT or SURF can to some extent solve these problems as they compute so-called point features, which are invariant towards scaling and rotation [8–11]. However, these algorithms are computationally complex and require powerful hardware in order to operate in real time. In automotive applications and generally in the field of mobile devices, limited processing power and the demand for low battery power consumption play an important role. Hence, adopting those sophisticated point feature algorithms to mobile hardware is an ambitious, but also necessary computer engineering task. This paper describes two embedded systems for SURF-based object recognition and pose estimation. The first system performs feature-based object recognition and has been implemented as a SoC on a single FPGA (Virtex-5 FX70). It can process images at a frame rate of up to five frames per second and (in our experiments) recognize and distinguish 9 different objects at a sensitivity of 91% and a specificity of 100% (no false positives). The second system determines the 3D pose of one or more objects. It features an ARM
References
[1]
H. M. Atiq, U. Farooq, R. Ibrahim, O. Khalid, and M. Amar, “Vehicle detection and shape recognition using optical sensors: a review,” in Proceedings of the 2nd International Conference on Machine Learning and Computing (ICMLC '10), pp. 223–227, February 2010.
[2]
C. Hermes, J. Einhaus, M. Hahn, C. W？hler, and F. Kummert, “Vehicle tracking and motion prediction in complex urban scenarios,” in Proceedings of the IEEE Intelligent Vehicles Symposium (IV '10), pp. 26–33, June 2010.
[3]
D. M. Jang and M. Turk, “Car-Rec: a real time car recognition system,” in Proceedings of the IEEE Workshop on Applications of Computer Vision (WACV '11), pp. 599–605, January 2011.
[4]
B. H？ferlin and K. Zimmermann, “Towards reliable traffic sign recognition,” in Proceedings of the IEEE Intelligent Vehicles Symposium, pp. 324–329, June 2009.
[5]
J. F. Liu, Y. F. Su, M. K. Ko, and P. N. Yu, “Development of a vision-based driver assistance system with lane departure warning and forward collision warning functions,” in Proceedings of the Digital Image Computing: Techniques and Applications (DICTA '08), pp. 480–485, December 2008.
[6]
S. J. Henderson and S. Feiner, “Evaluating the benefits of augmented reality for task localization in maintenance of an armored personnel carrier turret,” in Proceedings of the 8th IEEE International Symposium on Mixed and Augmented Reality (ISMAR '09), pp. 135–144, Orlando, Fla, USA, October 2009.
[7]
B. Schwerdtfeger, R. Reif, W. A. Günthner, et al., “Pick-by-vision: a first stress test,” in Proceedings of the 10th IEEE and ACM International Symposium on Mixed and Augmented Reality, pp. 115–124, October 2009.
[8]
G. Amato, F. Falchi, and P. Bolettieri, “Recognizing landmarks using automated classification techniques: an evaluation of various visual features,” in Proceedings of the 2nd International Conferences on Advances in Multimedia (MMEDIA '10), pp. 78–83, June 2010.
[9]
H. Bay, A. Ess, T. Tuytelaars, and L. Van Gool, “Speeded-up robust features (SURF),” Computer Vision and Image Understanding, vol. 110, no. 3, pp. 346–359, 2008.
[10]
C. Harris and M. Stevens, “A Combined corner and edge detector,” in Proceedings of the 4th Alvey Vision Conference, pp. 147–151, Manchester, UK, 1988.
[11]
D. G. Lowe, “Distinctive image features from scale-invariant keypoints,” International Journal of Computer Vision, vol. 60, no. 2, pp. 91–110, 2004.
[12]
X. Wu, Z. Shi, and Y. Zhong, “Detailed analysis and evaluation of keypoint extraction methods,” in Proceedings of the International Conference on Computer Application and System Modeling (ICCASM '10), vol. 2, pp. 562–566, October 2010.
[13]
M. Schaeferling and G. Kiefer, “Flex-SURF: a flexible architecture for FPGA-based robust feature extraction for optical tracking systems,” in Proceedings of the International Conference on Reconfigurable Computing and FPGAs (ReConFig '10), pp. 458–463, December 2010.
[14]
M. Schaeferling and G. Kiefer, “Object recognition on a chip: a complete SURF-based system on a single FPGA,” in Proceedings of the International Conference on Reconfigurable Computing and FPGAs (ReConFig '11), pp. 49–54, 2011.
[15]
K. Murphy, A. Torralba, D. Eaton, and W. Freeman, “Object detection and localization using local and global features,” in Toward Category-Level Object Recognition, vol. 4170, pp. 382–400, 2006.
[16]
A. Collet, M. Martinez, and S. S. Srinivasa, “The MOPED framework: object recognition and pose estimation for manipulation,” The International Journal of Robotics Research, vol. 30, no. 10, pp. 1284–1306, 2011.
[17]
F. Ren, J. Huang, R. Jiang, and R. Klette, “General traffic sign recognition by feature matching,” in Proceedings of the 24th International Conference Image and Vision Computing New Zealand (IVCNZ '09), pp. 409–414, November 2009.
[18]
D. Gossow, J. Pellenz, and D. Paulus, “Danger sign detection using color histograms and SURF matching,” in Proceedings of the IEEE International Workshop on Safety, Security and Rescue Robotics (SSRR '08), pp. 13–18, November 2008.
[19]
B. Besbes, A. Apatean, A. Rogozan, and A. Bensrhair, “Combining SURF-based local and global features for road obstacle recognition in far infrared images,” in Proceedings of the 13th International IEEE Conference on Intelligent Transportation Systems (ITSC '10), pp. 1869–1874, September 2010.
[20]
N. Cornells and L. V. Gool, “Fast scale invariant feature detection and matching on programmable graphics hardware,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops (CVPR '08), pp. 1–8, Anchorage, Alaska, USA, June 2008.
[21]
A. Schulz, F. Jung, S. Hartte, et al., “CUDA SURF—a real-time implementation for SURF,” http://www.d2.mpi-inf.mpg.de/surf.
[22]
P. Azad, T. Asfour, and R. Dillmann, “Combining Harris interest points and the SIFT descriptor for fast scale-invariant object recognition,” in Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS '09), pp. 4275–4280, October 2009.
[23]
W. C. Chen, Y. Xiong, J. Gao, N. Gelfand, and R. Grzeszczuk, “Efficient extraction of robust image features on mobile devices,” in Proceedings of the 6th IEEE and ACM International Symposium on Mixed and Augmented Reality (ISMAR '07), pp. 287–288, November 2007.
[24]
J. Zhang, H. Sang, and X. Shen, “Overview of approaches for accelerating scale invariant feature detection algorithm,” in Proceedings of the International Conference on Electric Information and Control Engineering (ICEICE '11), pp. 585–589, April 2011.
[25]
D. Bouris, A. Nikitakis, and I. Papaefstathiou, “Fast and efficient FPGA-based feature detection employing the SURF algorithm,” in Proceedings of the 18th IEEE International Symposium on Field-Programmable Custom Computing Machines (FCCM '10), pp. 3–10, May 2010.
[26]
J. ？váb, T. Krajník, J. Faigl, and L. P？eu？il, “FPGA based speeded up robust features,” in Proceedings of the IEEE International Conference on Technologies for Practical Robot Applications (TePRA '09), pp. 35–41, Woburn, Mass, USA, November 2009.
[27]
C. Evans, “Notes on the OpenSURF Library,” Tech. Rep. CSTR-09-001, University of Bristol, 2009.
[28]
V. Lepetit and P. Fua, “Monocular model-based 3D tracking of rigid objects: a survey,” Foundations and Trends in Computer Graphics and Vision, vol. 1, no. 1, 2005.
[29]
M. Muja and D. G. Lowe, “Fast approximate nearest neighbors with automatic algorithm configuration,” in Proceedings of the 4th International Conference on Computer Vision Theory and Applications (VISAPP '09), pp. 331–340, February 2009.
[30]
M. A. Fischler and R. C. Bolles, “Random sample consensus: a paradigm for model fitting with applications to image analysis and automated cartography,” Communications of the ACM, vol. 24, no. 6, pp. 381–395, 1981.
[31]
K. S. Arun, T. S. Huang, and S. D. Blostein, “Least-squares fitting of two 3-D point sets,” IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 9, no. 5, pp. 698–700, 1987.
