With technology advances and human requirements increasing, human-computer interaction plays an important role in our daily lives. Among these interactions, gesture-based recognition offers a natural and intuitive user experience that does not require physical contact and is becoming increasingly prevalent across various fields. Gesture recognition systems based on Frequency Modulated Continuous Wave (FMCW) millimeter-wave radar are receiving widespread attention due to their ability to operate without wearable sensors, their robustness to environmental factors, and the excellent penetrative ability of radar signals. This paper first reviews the current main gesture recognition applications. Subsequently, we introduce the system of gesture recognition based on FMCW radar and provide a general framework for gesture recognition, including gesture data acquisition, data preprocessing, and classification methods. We then discuss typical applications of gesture recognition systems and summarize the performance of these systems in terms of experimental environment, signal acquisition, signal processing, and classification methods. Specifically, we focus our study on four typical gesture recognition systems, including air-writing recognition, gesture command recognition, sign language recognition, and text input recognition. Finally, this paper addresses the challenges and unresolved problems in FMCW radar-based gesture recognition and provides insights into potential future research directions.
References
[1]
Zhou, Y., Jiang, G. and Lin, Y. (2016) A Novel Finger and Hand Pose Estimation Technique for Real-Time Hand Gesture Recognition. Pattern Recognition, 49, 102-114. https://doi.org/10.1016/j.patcog.2015.07.014
[2]
Gupta, H.P., Chudgar, H.S., Mukherjee, S., Dutta, T. and Sharma, K. (2016) A Continuous Hand Gestures Recognition Technique for Human-Machine Interaction Using Accelerometer and Gyroscope Sensors. IEEE Sensors Journal, 16, 6425-6432. https://doi.org/10.1109/jsen.2016.2581023
[3]
Liang, R. and Ouhyoung, M. (1995) A Real-Time Continuous Alphabetic Sign Language to Speech Conversion VR System. Computer Graphics Forum, 14, 67-76. https://doi.org/10.1111/j.1467-8659.1995.cgf143_0067.x
[4]
Kanokoda, T., Kushitani, Y., Shimada, M. and Shirakashi, J. (2019) Gesture Prediction Using Wearable Sensing Systems with Neural Networks for Temporal Data Analysis. Sensors, 19, Article 710. https://doi.org/10.3390/s19030710
[5]
Andrews, S. and Karthik, S. (2017) Burst Detection in Gesture Recognization for Emergency Communication Using Wearable Data Gloves. Proceedings of the 2017 Fourth HCT Information Technology Trends (ITT), Al Ain, 25-26 October 2017, 9-12. https://doi.org/10.1109/ctit.2017.8259558
[6]
Islam, M.R., Mitu, U.K., Bhuiyan, R.A. and Shin, J. (2018) Hand Gesture Feature Extraction Using Deep Convolutional Neural Network for Recognizing American Sign Language. Proceedings of the 2018 4th International Conference on Frontiers of Signal Processing (ICFSP), Poitiers, 24-27 September 2018, 115-119. https://doi.org/10.1109/icfsp.2018.8552044
[7]
Plouffe, G. and Cretu, A.-M. (2016) Static and Dynamic Hand Gesture Recognition in Depth Data Using Dynamic Time Warping. IEEE Transactions on Instrumentation and Measurement, 65, 305-316. https://doi.org/10.1109/tim.2015.2498560
[8]
Li, Y., Miao, Q., Tian, K., Fan, Y., Xu, X., Li, R., et al. (2018) Large-Scale Gesture Recognition with a Fusion of RGB-D Data Based on Saliency Theory and C3D Model. IEEE Transactions on Circuits and Systems for Video Technology, 28, 2956-2964. https://doi.org/10.1109/tcsvt.2017.2749509
[9]
Wang, Z., Jiang, K., Hou, Y., Dou, W., Zhang, C., Huang, Z., et al. (2019) A Survey on Human Behavior Recognition Using Channel State Information. IEEE Access, 7, 155986-156024. https://doi.org/10.1109/access.2019.2949123
[10]
Wang, Z., Song, X., Fan, J., Chen, F., Zhou, N., Guo, Y., et al. (2021) WiDG: An Air Hand Gesture Recognition System Based on CSI and Deep Learning. Proceedings of the 2021 33rd Chinese Control and Decision Conference (CCDC), Kunming, 22-24 May 2021, 1243-1248. https://doi.org/10.1109/ccdc52312.2021.9602438
[11]
Pu, Q., Gupta, S., Gollakota, S. and Patel, S. (2013) Whole-Home Gesture Recognition Using Wireless Signals. Proceedings of the 19th Annual International Conference on Mobile Computing & Networking, Miami, 30 September-4 October 2013, 27-38. https://doi.org/10.1145/2500423.2500436
[12]
Choi, J.-W., Ryu, S.-J. and Kim, J.-H. (2019) Short-Range Radar Based Real-Time Hand Gesture Recognition Using LSTM Encoder. IEEE Access, 7, 33610-33618. https://doi.org/10.1109/access.2019.2903586
[13]
Yang, Z. and Zheng, X. (2021) Hand Gesture Recognition Based on Trajectories Features and Computation-Efficient Reused LSTM Network. IEEE Sensors Journal, 21, 16945-16960. https://doi.org/10.1109/jsen.2021.3079564
[14]
Wang, Z., Liu, F., Li, X., Ma, M., Feng, X. and Guo, Y. (2022) A Survey of Hand Gesture Recognition Based on FMCW Radar. Proceedings of the 8th International Conference on Communication and Information Processing, Beijing, 3-5 November 2022, 73-79. https://doi.org/10.1145/3571662.3571674
[15]
Ullmann, I., Guendel, R.G., Kruse, N.C., Fioranelli, F. and Yarovoy, A. (2023) A Survey on Radar-Based Continuous Human Activity Recognition. IEEE Journal of Microwaves, 3, 938-950. https://doi.org/10.1109/jmw.2023.3264494
[16]
Wang, P., Lin, J., Wang, F., Xiu, J., Lin, Y., Yan, N., et al. (2020) A Gesture Air-Writing Tracking Method that Uses 24 GHz SIMO Radar SoC. IEEE Access, 8, 152728-152741. https://doi.org/10.1109/access.2020.3017869
[17]
Zhang, Z., Tian, Z. and Zhou, M. (2018) Latern: Dynamic Continuous Hand Gesture Recognition Using FMCW Radar Sensor. IEEE Sensors Journal, 18, 3278-3289. https://doi.org/10.1109/jsen.2018.2808688
[18]
Li, G., Zhang, R., Ritchie, M. and Griffiths, H. (2018) Sparsity-Driven Micro-Doppler Feature Extraction for Dynamic Hand Gesture Recognition. IEEE Transactions on Aerospace and Electronic Systems, 54, 655-665. https://doi.org/10.1109/taes.2017.2761229
[19]
Wang, S., Song, J., Lien, J., Poupyrev, I. and Hilliges, O. (2016) Interacting with Soli: Exploring Fine-Grained Dynamic Gesture Recognition in the Radio-Frequency Spectrum. Proceedings of the 29th Annual Symposium on User Interface Software and Technology, Tokyo, 16-19 October 2016, 851-860. https://doi.org/10.1145/2984511.2984565
[20]
Wang, Y., Wang, S., Zhou, M., Jiang, Q. and Tian, Z. (2019) TS-I3D Based Hand Gesture Recognition Method with Radar Sensor. IEEE Access, 7, 22902-22913. https://doi.org/10.1109/access.2019.2897060
[21]
Ryu, S.-J., Suh, J.-S., Baek, S.-H., Hong, S. and Kim, J.-H. (2018) Feature-Based Hand Gesture Recognition Using an FMCW Radar and Its Temporal Feature Analysis. IEEE Sensors Journal, 18, 7593-7602. https://doi.org/10.1109/jsen.2018.2859815
[22]
Wang, Y., Ren, A., Zhou, M., Wang, W. and Yang, X. (2020) A Novel Detection and Recognition Method for Continuous Hand Gesture Using FMCW Radar. IEEE Access, 8, 167264-167275. https://doi.org/10.1109/access.2020.3023187
[23]
Malysa, G., Wang, D., Netsch, L. and Ali, M. (2016) Hidden Markov Model-Based Gesture Recognition with FMCW Radar. Proceedings of the 2016 IEEE Global Conference on Signal and Information Processing (GlobalSIP), Washington DC, 7-9 December 2016, 1017-1021. https://doi.org/10.1109/globalsip.2016.7905995
[24]
Wan, Q., Li, Y., Li, C. and Pal, R. (2014) Gesture Recognition for Smart Home Applications Using Portable Radar Sensors. Proceedings of the 2014 36th Annual International Conference of the IEEE Engineering in Medicine and Biology Society, Chicago, 26-30 August 2014, 6414-6417.
[25]
Lien, J., Gillian, N., Karagozler, M.E., Amihood, P., Schwesig, C., Olson, E., et al. (2016) Soli: Ubiquitous Gesture Sensing with Millimeter Wave Radar. ACM Transactions on Graphics, 35, 1-19. https://doi.org/10.1145/2897824.2925953
[26]
Dekker, B., Jacobs, S., Kossen, A.S., Kruithof, M.C., Huizing, A.G. and Geurts, M. (2017) Gesture Recognition with a Low Power FMCW Radar and a Deep Convolutional Neural Network. Proceedings of the 2017 European Radar Conference (EURAD), Nuremberg, 11-13 October 2017, 163-166. https://doi.org/10.23919/eurad.2017.8249172
[27]
Liu, D. and Meng, H. (2020) Application of FMCW Radar for the Recongnition of Hand Gesture Using Time Series Convolutional Neural Networks. Proceedings of the 2020 International Conference on Microwave and Millimeter Wave Technology (ICMMT), Shanghai, 20-23 September 2020, 1-3. https://doi.org/10.1109/icmmt49418.2020.9386392
[28]
Hazra, S. and Santra, A. (2018) Robust Gesture Recognition Using Millimetric-Wave Radar System. IEEE Sensors Letters, 2, 1-4. https://doi.org/10.1109/lsens.2018.2882642
[29]
Regani, S.D., Wu, C., Wang, B., Wu, M. and Liu, K.J.R. (2021) mmWrite: Passive Handwriting Tracking Using a Single Millimeter-Wave Radio. IEEE Internet of Things Journal, 8, 13291-13305. https://doi.org/10.1109/jiot.2021.3066507
[30]
Liu, H., Zhou, A., Dong, Z., Sun, Y., Zhang, J., Liu, L., et al. (2022) M-Gesture: Person-Independent Real-Time In-Air Gesture Recognition Using Commodity Millimeter Wave Radar. IEEE Internet of Things Journal, 9, 3397-3415. https://doi.org/10.1109/jiot.2021.3098338
[31]
Zhang, Z., Tian, Z., Zhou, M., Nie, W. and Li, Z. (2018) Riddle: Real-Time Interacting with Hand Description via Millimeter-Wave Sensor. Proceedings of the 2018 IEEE International Conference on Communications (ICC), Kansas City, 20-24 May 2018, 1-6. https://doi.org/10.1109/icc.2018.8422765
[32]
Rahman, M.M., Mdrafi, R., Gurbuz, A.C., Malaia, E., Crawford, C., Griffin, D., et al. (2021) Word-Level Sign Language Recognition Using Linguistic Adaptation of 77 GHz FMCW Radar Data. Proceedings of the 2021 IEEE Radar Conference (RadarConf21), Atlanta, 7-14 May 2021, 1-6. https://doi.org/10.1109/radarconf2147009.2021.9455190
[33]
Santhalingam, P.S., Hosain, A.A., Zhang, D., Pathak, P., Rangwala, H. and Kushalnagar, R. (2020) mmASL: Environment-Independent ASL Gesture Recognition Using 60 GHz Millimeter-Wave Signals. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 4, 1-30. https://doi.org/10.1145/3381010
[34]
Hu, Y., Wang, B., Wu, C. and Liu, K.J.R. (2022) mmKey: Universal Virtual Keyboard Using a Single Millimeter-Wave Radio. IEEE Internet of Things Journal, 9, 510-524. https://doi.org/10.1109/jiot.2021.3084560
[35]
Wei, H., Li, Z., Galvan, A.D., Su, Z., Zhang, X., Pahlavan, K., et al. (2022) IndexPen: Two-Finger Text Input with Millimeter-Wave Radar. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 6, 1-39. https://doi.org/10.1145/3534601
[36]
Zhang, Z., Tian, Z., Zhang, Y., Zhou, M. and Wang, B. (2019) u-DeepHand: FMCW Radar-Based Unsupervised Hand Gesture Feature Learning Using Deep Convolutional Auto-Encoder Network. IEEE Sensors Journal, 19, 6811-6821. https://doi.org/10.1109/jsen.2019.2910810
