With the wide applications of vision based intelligent systems, image and video analysis technologies have attracted the attention of researchers in the computer vision field. In image and video analysis, human activity recognition is an important research direction. By interpreting and understanding human activity, we can recognize and predict the occurrence of crimes and help the police or other agencies react immediately. In the past, a large number of papers have been published on human activity recognition in video and image sequences. In this paper, we provide a comprehensive survey of the recent development of the techniques, including methods, systems, and quantitative evaluation towards the performance of human activity recognition.
References
[1]
Lacko, D. Motion Capture and Guidance Using Open Source Hardware. Master Thesis, Artesis University College of Antwerp, Antwerp, Belgium, 2011.
[2]
Kautz, H. A. Formal Theory of Plan Recognition. Ph.D. Thesis, University of Rochester, New York, NY, USA, 1987.
[3]
Sheikh, Y.; Shah, M. Bayesian modeling of dynamic scenes for object detection. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 1778–1792.
[4]
Yilmaz, A.; Javed, O.; Shah, M. Object tracking: A survey. ACM Comput Surv. 2006, 38, art no. 13.
[5]
Ko, T.; Shah, M. A survey on behavior analysis in video surveillance for homeland security applications. Proceedings of Workshop on Applied Imagery Pattern Recognition, Washington, DC, USA, 15–17 October 2008; pp. 1–8.
[6]
Lavee, G.; Rivlin, E.; Rudzsky, M. Understanding Video Events: A Survey of Methods for Automatic Interpretation of Semantic Occurrences in Video. IEEE Trans. Syst. Man Cybern Part C 2009, 39, 489–504.
[7]
Popoola, O.P.; Wang, K. Video-Based Abnormal Human Behavior Recognition—A Review. IEEE Trans. Syst. Man Cybern Part C 2012, 42, 865–878.
[8]
Liao, L. Location-Based Activity Recognition. Ph.D. Thesis, Department of Computer Science and Engineering, University of Washington, Washington, DC, USA, 2006.
[9]
Aggarwal, J.K.; Ryoo, M.S. Human activity analysis: A review. ACM Comput. Surv. 2011, 43, art no. 16.
[10]
Casdagli, M.; Eubank, S.; Farmer, J.D.; Gibson, J. State space reconstruction in the presence of noise. Physica D 1991, 51, 52–98.
[11]
Bobick, A.; Davis, J. The recognition of human movement using temporal templates. IEEE Trans. Pattern Anal. Mach. Intell. 2001, 23, 257–267.
[12]
Oren, M.; Papageorgiou, C.; Sinha, P.; Osuna, E.; Poggio, T. Pedestrian detection using wavelet templates. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Juan, Argentina, 17–19 June 1997; pp. 193–199.
[13]
Ben-Arie, J.; Wang, Z.; Pandit, P.; Rajaram, S. Human activity recognition using multidimensional indexing. IEEE Trans. Pattern Anal. Mach. Intell. 2002, 24, 1091–1104.
[14]
Lu, W.; Okuma, K.; Little, J. Tracking and recognizing actions of multiple hockey players using the boosted particle filter. Image Vis. Comput. 2009, 27, 189–205.
[15]
Khalid, S.; Naftel, A. Classifying spatiotemporal object trajectories using unsupervised learning of basis function coefficients. Proceedings of the 3rd ACM International Workshop on Video Surveillance & Sensor Networks, New York, NY, USA, 1–2 August 2005; pp. 45–52.
[16]
Wilson, A.D.; Bobick, A.F. Recognition and interpretation of parametric gesture. Proceedings of IEEE International Conference on Computer Vision,, Bombay, India, 4–7 January 1998; pp. 329–336.
[17]
Duong, T.; Bui, H.; Phung, D.; Venkatesh, S. Activity recognition and abnormality detection with the switching hidden semi-Markov model. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition, San Diego, CA, USA, 20–26 June 2005; pp. 838–845.
[18]
Duong, T.; Phung, D.; Bui, H.; Venkatesh, S. Efficient duration and hierarchical modeling for human activity recognition. Artif. Intell. 2009, 173, 830–856.
[19]
Chieu, H.; Lee, W.; Kaelbling, L. Activity recognition from physiological data using conditional random fields. Tech. Rep. Singapore MIT Alliance Symp. 2006.
[20]
Yin, J.; Hu, D.; Yang, Q. Spatio-temporal event detection using dynamic conditional random fields. Proceedings of the International Joint Conferences on Artificial Intelligence, Pasadena, CA, USA, 11–17 July 2009; pp. 1321–1326.
[21]
Yin, J.; Meng, Y. Abnormal behavior recognition using self-adaptive hidden markov models. Lect. Notes Comput. Sci. 2009, 5627, 337–346.
[22]
Loy, C.C.; Xiang, T.; Gong, S. Surveillance video behaviour profiling and anomaly detection. Proc. SPIE 2009, 7486, 74860E.
[23]
Hu, D.H.; Yang, Q. CIGAR: Concurrent and interleaving goal and activity recognition. Proceedings of the National Conference on Artificial Intelligence, Chicago, IL, USA, 13–17 July 2008; pp. 1363–1368.
[24]
Wang, L.; Hu, W.; Tan, T. Recent developments in human motion analysis. Pattern Recognit 2003, 36, 585–601.
[25]
Xiang, T.; Gong, S. Video behavior profiling for anomaly detection. IEEE Trans. Pattern Anal. Mach. Intell. 2008, 30, 893–908.
[26]
Robertson, N.; Reid, I. A general method for human activity recognition in video. Comput. Vis. Image Underst. 2006, 104, 232–248.
[27]
Bishop, C.M. Pattern Recognition and Machine Learning; Springer: New York, NY, USA, 2006.
[28]
Vail, D.L.; Veloso, M.; Lafferty, J.D. Conditional random fields for activity recognition. Proceedings of the 6th international joint conference on Autonomous agents and multiagent systems, Honolulu, HI, USA, 14–18 May 2007; p. art. no. 235.
[29]
Lui, Y.; Beveridge, J.R.; Kirby, M. Action classification on product manifolds. Proceedings of IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR), San Francisco, CA, USA, 13– 18 June 2010; pp. 833–839.
[30]
Harandi, M.T.; Sanderson, C.; Wiliem, A.; Lovell, B.C. Kernel analysis over riemannian manifolds for visual recognition of actions, pedestrians and textures. Proceedings of IEEE Workshop on Applications of Computer Vision (WACV), Breckenridge, CO, USA, 9– 11 January 2012; pp. 433–439.
[31]
Lui, Y. Advances in Matrix Manifolds for Computer Vision. Image Vision Comput. 2012, 30, 380–388.
[32]
Shin, J.; Kim, S.; Kang, S.; Lee, S.; Paik, J.; Abidi, B.; Abidi, M. Optical flow-based real-time object tracking using non-prior training active feature model. Real Time Imaging 2005, 11, 204–218.
[33]
Amer, A.; Regazzoni, C. Introduction to the special issue on video object processing for surveillance applications. Real Time Imaging 2005, 11, 167–171.
[34]
Kumar, P.; Mittal, A.; Kumar, P. Study of robust and intelligent surveillance in visible and multi-modal framework. Informatica 2008, 32, 63–77.
[35]
Nwagboso, C. User focused surveillance systems integration for intelligent transport systems. In Advanced Video-based Surveillance Systems; Kluwer Academic Publishers: Boston, MA, USA, 1998. Chapter 1.1; pp. 8–12.
[36]
Wren, C.R.; Azarbayejani, A.; Darrell, T.; Pentland, A.P. Pfinder: Real-time tracking of the human body. IEEE Trans. Pattern Anal. Mach. Intell. 1997, 19, 780–785.
[37]
Haritaoglu, I.; Harwood, D.; Davis, L.S. W4: Real-time surveillance of people and their activities. IEEE Trans. Pattern Anal. Mach. Intell. 2000, 22, 809–830.
[38]
Toole, A.J.; Harms, J.; Snow, S.L. A video database of moving faces and people. IEEE Trans. Pattern Anal. Mach. Intell. 2005, 27, 812–816.
[39]
List, T.; Bins, J.; Fisher, R.B.; Tweed, D.; Thorisson, K.R. Two approaches to a plug-and-play vision architecture—CAVIAR and psyclone. Proceedings of AAAI Workshop on Modular Construction of Human-Like Intelligence, Pittsburgh, PA, USA, 10 July 2005; pp. 16–23.
[40]
Tweed, D.; Fang, W.; Fisher, R.; Bins, J.; List, T. Exploring techniques for behavior recognition via the CAVIAR modular vision framework. Proceedings of Workshop on Human Activity Recognition and Modeling, Oxford, UK, October 2005; pp. 97–104.
[41]
Andrade, E.L.; Blunsden, S.; Fisher, R.B. Modelling Crowd Scenes for Event Detection. Proceedings of 18th International Conference on Pattern Recognition, Hong Kong, China, 20–24 August 2006; pp. 175–178.
[42]
Collins, R.T.; Lipton, A.J.; Kanade, T.; Fujiyoshi, H.; Duggins, D.; Tsin, Y.; Tolliver, D.; Enomoto, N.; Hasegawa, O.; Burt, P.; et al. A System for Video Surveillance and Monitoring: VSAM Final Report. CMU-RI-TR-00-12, Technical Report; Carnegie Mellon University: Pittsburgh, PA, USA, 2000.
[43]
Wang, L.; Tan, T.; Ning, H.; Hu, W. Fusion of Static and Dynamic Body Biometrics for Gait Recognition. IEEE Trans Circuits Syst. Video Technol. 2004, 14, 149–158.
[44]
Tian, Y.; Brown, L.; Hampapur, A.; Lu, M.; Senior, A.; Shu, C. IBM smart surveillance system (S3): Event based video surveillance system with an open and extensible framework. Mach. Vis. Appl. 2008, 19, 315–327.
[45]
Kasturi, R.; Goldgof, D.; Soundararajan, P.; Manohar, V.; Boonstra, M.; Korzhova, V. Performance Evaluation Protocol for Text, Face, Hands, Person and Vehicle Detection & Tracking in Video Analysis and Content Extraction (VACE-II). Technical Report; University of South Florida: Tampa, FL, USA, 2005.
[46]
Manohar, V.; Soundararajan, P.; Raju, H.; Goldgof, D.; Kasturi, R.; Garofolo, J. Performance evaluation of object detection and tracking in video. Proceedings of the Seventh Asian Conference on Computer Vision, Hyderabad, India, 13–16 January 2006; pp. 151–161.
[47]
Raju, H.; Prasad, S.; Sharma, P. Annotation Guidelines for Video Analysis and Content Extraction (VACE-II). Technical Report; Video Mining Inc.: Tampa, FL, USA, 2006.
[48]
Collins, R.; Zhou, X.; Teh, S.K. An open source tracking testbed and evaluation web site. Proceedings of IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Beijing, China, 15–16 October 2005; pp. 17–24.
[49]
Smeaton, A.F.; Over, P.; Kraaij, W. Evaluation campaigns and TRECVid. Proceedings of the 8th ACM International Workshop on Multimedia Information Retrieval, Santa Barbara, CA, USA, 26–27 October 2006; pp. 321–330.
[50]
Nghiem, A.T.; Bremond, F.; Thonnat, M.; Valentin, V. ETISEO, performance evaluation for video surveillance systems. Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance, London, UK, 5–7 September 2007; pp. 476–481.
[51]
Brown, L.M.; Senior, A.W.; Tian, Y.; Connell, J.; Hampapur, A.; Shu, C.; Merkl, H.; Lu, M. Performance evaluation of surveillance systems under varying conditions. Proceedings of IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, Beijing, China, 15–16 October 2005; pp. 79–87.
[52]
Young, D.; Ferryman, J. PETS metrics: On-line performance evaluation service. Proceedings of Joint IEEE Workshop on Visual Surveillance and Performance Evaluation of Tracking and Surveillance, Beijing, China, 15–16 October 2005; pp. 317–324.
[53]
Kasturi, R.; Goldgof, D.; Soundararajan, P.; Manohar, V.; Garofolo, J.; Bowers, R.; Boonstra, M.; Korzhova, V.; Zhang, J. Framework for performance evaluation of face, text, and vehicle detection and tracking in video: Data, metrics, and protocol. IEEE Trans. Pattern Anal. Mach. Intell. 2009, 31, 319–336.
[54]
Stiefelhagen, R.; Steusloff, H.; Waibel, A. CHIL: Computers in the human interaction loop. Proceedings of Workshop on Image Analysis for Multimedia Interactive Services, Lisbon, Portugal, 21–23 April 2004.
[55]
Ziliani, F.; Velastin, S.; Porikli, F.; Marcenaro, L.; Kelliher, T.; Cavallaro, A.; Bruneaut, P. Performance evaluation of event detection solutions: The CREDS experience. Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance, Como, Italy, 15–16 September 2005; pp. 201–206.
[56]
Desurmont, X.; Carincotte, C.; Bremond, F. Intelligent video systems: A review of performance evaluation metrics that use mapping procedures. Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance, Boston, MA, USA, 29 August–1 September 2010; pp. 127–134.
[57]
Mostefa, D.; Moreau, N.; Choukri, K.; Potamianos, G.; Chu, S.M.; Tyagi, A.; Casas, J.R.; Turmo, J.; Cristoforetti, L.; Tobia, F.; et al. The CHIL audiovisual corpus for lecture and meeting analysis inside smart rooms. Lang. Resour. Eval. 2007, 41, 389–407.
[58]
Manohar, V.; Boonstra, M.; Korzhova, V.; Soundararajan, P.; Goldgof, D.; Kasturi, R. PETS vs. VACE evaluation programs: A comparative study. Proceedings of IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, New York, NY, USA, 18 June 2006; pp. 1–6.
[59]
Ellis, A.; Ferryman, J. PETS2010 and PETS2009 evaluation of results using individual ground truthed single views. Proceedings of IEEE International Conference on Advanced Video and Signal Based Surveillance, Boston, MA, USA, 29 August–1 September 2010; pp. 135–142.
[60]
Desurmont, X.; Sebbe, R.; Martin, F.; Machy, C.; Delaigle, J.F. Performance evaluation of frequent events detection systems. Proceedings of IEEE International Workshop on Performance Evaluation of Tracking and Surveillance, New York, NY, USA, 18 June 2006; pp. 15–21.
