Phishing is a form of cybercrime in which people are deceived into exposing their personal information which can result in financial loss. These attacks are often executed via fraudulent messages, misleading advertisements and compromised legitimate websites. This study proposes a framework based on Quantile Regression Deep Q-Network (QR-DQN) that integrates RoBERTa semantic embeddings and crafted lexical features to enhance phishing detection. Instead of predicting mean returns, QR-DQN uses quantile regression to model the distribution over returns which improves stability and generalization for previously unseen phishing samples over traditional RL DQN approaches when combined with semantic embeddings. A custom crawled diverse dataset of 105,000 URLs were curated from PhishTank, OpenPhish, Cloudflare etc. The framework uses an 80/20 split of the dataset. The QR-DQN model with RoBERTa embeddings and lexical features achieved test accuracy 99.86%, precision 99.75%, recall 99.96% and F1-score 99.85% demonstrating high effectiveness. Compared to the standard DQN with lexical features, the suggested QR-DQN framework with lexical and semantic features lowers the generalization gap from 1.66 to 0.04 percent. The experiments using 5-fold cross-validation have resulted in consistent results under this protocol with a mean accuracy of 99.90% and standard deviation of 0.04%. This shows the hybrid technique which combines quantile-based value estimation with RoBERTa semantic embeddings and lexical features reports strong performance and reduced generalization gap.
References
[1]
Putra, F.P.E., Ubaidi, U., Zulfikri, A., Arifin, G. and Ilhamsyah, R.M. (2024) Analysis of Phishing Attack Trends, Impacts and Prevention Methods: Literature Study. Brilliance: Research of Artificial Intelligence, 4, 413-421. https://doi.org/10.47709/brilliance.v4i1.4357
[2]
Akbar, N. (2014) Analysing Persuasion Principles in Phishing Emails. Ph.D. Thesis, University of Twente.
[3]
Ejaz, A., Mian, A.N. and Manzoor, S. (2023) Life-Long Phishing Attack Detection Using Continual Learning. Scientific Reports, 13, Article No. 11488. https://doi.org/10.1038/s41598-023-37552-9
[4]
Nguyen, T.T. and Reddi, V.J. (2023) Deep Reinforcement Learning for Cyber Security. IEEE Transactions on Neural Networks and Learning Systems, 34, 3779-3795. https://doi.org/10.1109/tnnls.2021.3121870
[5]
Sarker, I.H. (2021) Deep Cybersecurity: A Comprehensive Overview from Neural Network and Deep Learning Perspective. SN Computer Science, 2, Article No. 154. https://doi.org/10.1007/s42979-021-00535-6
[6]
Su, M. and Su, K. (2023) Bert-Based Approaches to Identifying Malicious URLs. Sensors, 23, Article 8499. https://doi.org/10.3390/s23208499
[7]
Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2015) Human-Level Control through Deep Reinforcement Learning. Nature, 518, 529-533. https://www.nature.com/articles/nature14236
[8]
Liu, R., Wang, Y., Xu, H., Qin, Z., Zhang, F., Liu, Y., et al. (2025) PMANet: Malicious URL Detection via Post-Trained Language Model Guided Multi-Level Feature Attention Network. Information Fusion, 113, Article 102638. https://doi.org/10.1016/j.inffus.2024.102638
[9]
AVS Kumar, S., et al. (2024) Phishing Email Detection Using Machine Learning. International Journal of Artificial Intelligence and Data Analysis, 11, 48-59.
[10]
Rao, R.S., Kondaiah, C., Pais, A.R. and Lee, B. (2025) A Hybrid Super Learner Ensemble for Phishing Detection on Mobile Devices. Scientific Reports, 15, Article 16308. https://doi.org/10.1038/s41598-025-02009-8
[11]
Chatterjee, M. and Namin, A. (2019) Detecting Phishing Websites through Deep Reinforcement Learning. 2019 IEEE 43rd Annual Computer Software and Applications Conference (COMPSAC), Milwaukee, 15-19 July 2019, 227-232. https://doi.org/10.1109/compsac.2019.10211
[12]
Lopez-Martin, M., Carro, B. and Sanchez-Esguevillas, A. (2020) Application of Deep Reinforcement Learning to Intrusion Detection for Supervised Problems. Expert Systems with Applications, 141, Article 112963. https://doi.org/10.1016/j.eswa.2019.112963
[13]
Terranova, F., et al. (2024) Leveraging Deep Reinforcement Learning for Cyber-Attack Path Discovery. ACM Digital Library.
[14]
Sahingoz, O.K., Buber, E., Demir, O. and Diri, B. (2019) Machine Learning Based Phishing Detection from URLs. Expert Systems with Applications, 117, 345-357. https://doi.org/10.1016/j.eswa.2018.09.029
[15]
Kheddar, H., Dawoud, D.W., Awad, A.I., Himeur, Y. and Khan, M.K. (2024) Reinforcement-Learning-Based Intrusion Detection in Communication Networks: A Review. IEEE Open Journal of the Communications Society, 5, 2115-2141.
[16]
Devlin, J., Chang, M., Lee, K. and Toutanova, K. (2019) BERT: Pre-Training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North, Minneapolis, 2 June-7 June 2019, 4171-4186. https://doi.org/10.18653/v1/n19-1423
[17]
Young, T., Hazarika, D., Poria, S. and Cambria, E. (2018) Recent Trends in Deep Learning Based Natural Language Processing [Review Article]. IEEE Computational Intelligence Magazine, 13, 55-75. https://doi.org/10.1109/mci.2018.2840738
[18]
Maneriker, P., Stokes, J.W., Lazo, E.G., Carutasu, D., Tajaddodianfar, F. and Gururajan, A. (2021). URLTran: Improving Phishing URL Detection Using Transformers. MILCOM 2021-2021 IEEE Military Communications Conference (MILCOM), San Diego, 29 November-2 December 2021, 197-204. https://doi.org/10.1109/milcom52596.2021.9653028
[19]
Saleem Raja, A., Vinodini, R. and Kavitha, A. (2021) Lexical Features Based Malicious URL Detection Using Machine Learning Techniques. Materials Today: Proceedings, 47, 163-166. https://doi.org/10.1016/j.matpr.2021.04.041
[20]
Ari Kustiawan, Y. and Ghauth, K.I. (2025) Evaluating the Impact of Feature Engineering in Phishing URL Detection: A Comparative Study of URL, HTML, and Derived Features. IEEE Access, 13, 126756-126768. https://doi.org/10.1109/access.2025.3579223
[21]
Asif, A.U.Z., Shirazi, H. and Ray, I. (2023) Machine Learning-Based Phishing Detection Using URL Features: A Comprehensive Review. In: Dolev, S. and Schieber, B., Eds., Lecture Notes in Computer Science, Springer, 481-497. https://doi.org/10.1007/978-3-031-44274-2_36
Luong, N.C., Hoang, D.T., Gong, S., Niyato, D., Wang, P., Liang, Y., et al. (2019) Applications of Deep Reinforcement Learning in Communications and Networking: A Survey. IEEE Communications Surveys & Tutorials, 21, 3133-3174. https://doi.org/10.1109/comst.2019.2916583
[24]
Mnih, V., Kavukcuoglu, K., Silver, D., et al. (2013) Playing Atari with Deep Reinforcement Learning. arXiv:1312.5602. https://arxiv.org/abs/1312.5602
[25]
Kim, T., Park, N., Hong, J. and Kim, S. (2022) Phishing URL Detection: A Net-Work-Based Approach Robust to Evasion. Proceedings of the 2022 ACM SIGSAC Conference on Computer and Communications Security, Los Angeles, 7-11 November 2022, 1679-1782. https://doi.org/10.1145/3548606.3560615
[26]
Otieno, D.O., Abri, F., Namin, A.S. and Jones, K.S. (2023) Detecting Phishing URLs using the BERT Transformer Model. 2023 IEEE International Conference on Big Data (BigData), Sorrento, 15-18 December 2023, 1303-1310.
[27]
Sutton, R.S. and Barto, A.G. (2018) Reinforcement Learning: An Introduction. 2nd Edition, MIT Press.
[28]
Dabney, W., Rowland, M., Bellemare, M. and Munos, R. (2018) Distributional Reinforcement Learning with Quantile Regression. Proceedings of the AAAI Conference on Artificial Intelligence, 32, 2892-2901. https://doi.org/10.1609/aaai.v32i1.11791
[29]
Bellemare, M.G., Dabney, W. and Munos, R. (2017) A Distributional Perspective on Reinforcement Learning. Proceedings of the 34th International Conference on MachineLearning, Sydney, 6-11 August 2017, 449-458.
[30]
Maci, A., Santorsola, A., Coscia, A. and Iannacone, A. (2023) Unbalanced Web Phishing Classification through Deep Reinforcement Learning. Computers, 12, Article 118. https://doi.org/10.3390/computers12060118
[31]
Egigogo, O.E., Idris, I.A., Olalere, O.M., Abisoye, O.G. and Ojeniyi, B.A. (2022) Development of Hybridized CNN-BiGRU Framework for Detection of Website Phishing Attacks. Nigerian Journal of Technological Research, 3, 45-54.
