In light of the ongoing challenge posed by unknown vulnerabilities in software solutions, the prompt identification and rectification of these weaknesses are of vital importance. Traditional penetration testing, although thorough, typically demands substantial human resources and time commitments. To contend with this issue, we propose an innovative penetration testing framework that incorporates natural language processing, utilizing large language model (LLM) driven multi-agent systems to automate and expedite vulnerability detection. By integrating insights from established exploit signatures and AI-driven fuzzing techniques, our framework, LLMAZD, preemptively examines target software within a controlled setting, effectively identifying both zero-day and one-day exploits, as well as system vulnerabilities, in a cost-effective manner. The framework demonstrates versatility in detecting multiple vulnerability classes, including SQL injection, Cross-Site Scripting (XSS), Cross-Site Request Forgery (CSRF), file upload vulnerabilities, and other different types, all while maintaining efficiency and cost-effectiveness throughout the testing process. Extensive evaluations on more than 20 CVE test cases reveal that the framework substantially enhances detection capabilities, reduces the overall burden of testing, and offers a practical pathway towards more secure software ecosystems.
References
[1]
Mandiant and Google’s TAG (2024) A Year in Review of Zero-Days Exploited In-the-Wild in 2023. https://storage.googleapis.com/gweb-uniblog-publish-prod/documents/Year_in_Review_of_ZeroDays.pdf
[2]
Zhu, Y., Kellermann, A., Gupta, A., Li, P., Fang, R., Bindu, R. and Kang, D. (2025) Teams of LLM Agents Can Exploit Zero-Day Vulnerabilities. arXiv:2406.01637. https://arxiv.org/abs/2406.01637
[3]
Shen, X., Wang, L., Li, Z., Chen, Y., Zhao, W., Sun, D., Wang, J. and Ruan, W. (2024) PentestAgent: Incorporating LLM Agents to Automated Penetration Testing. arXiv: 2411.05185. https://arxiv.org/abs/2411.05185
[4]
Sood, A.K., Zeadally, S. and Hong, E. (2025) The Paradigm of Hallucinations in Ai-Driven Cybersecurity Systems: Understanding Taxonomy, Classification Outcomes, and Mitigations. Computers and Electrical Engineering, 124, Article 110307. https://doi.org/10.1016/j.compeleceng.2025.110307
[5]
Google’s Big Sleep Team (2024) From Naptime to Big Sleep: Using Large Language Models to Catch Vulnerabilities in Real-World Code. https://googleprojectzero.blog-spot.com/2024/10/from-naptime-to-big-sleep.html
[6]
Bianou, S.G. and Batogna, R.G. (2024) PENTEST-AI, an LLM-Powered Multi-Agents Framework for Penetration Testing Automation Leveraging MITRE Attack. 2024 IEEE International Conference on Cyber Security and Resilience (CSR), London, 2-4 September 2024, 763-770. https://doi.org/10.1109/csr61664.2024.10679480
[7]
Deng, G., Liu, Y., Mayoral-Vilches, V., Liu, P., Li, Y., Xu, Y., Zhang, T., Liu, Y., Pinzger, M. and Rass, S. (2024) PentestGPT: An LLM-Empowered Automatic Penetration Testing Tool. arXiv:2308.06782. https://arxiv.org/abs/2308.06782
[8]
Moskal, S., Laney, S., Hemberg, E. and O’Reilly, U.-M. (2023) LLMS Killed the Script kiddie: How Agents Supported by Large Language Models Change the Landscape of Network Threat Testing. arXiv:2310.06936. https://arxiv.org/abs/2310.06936
[9]
Touré, A., Imine, Y., Semnont, A., Delot, T. and Gallais, A. (2024) A Framework for Detecting Zero-Day Exploits in Network Flows. Computer Networks, 248, Article 110476. https://doi.org/10.1016/j.comnet.2024.110476
[10]
Teymourlouei, H., Stone, D. and Jackson, L. (2023) Identifying Zero-Day Attacks with Machine Learning and Data Reduction Methods. 2023 Congress in Computer Science, Computer Engineering, & Applied Computing (CSCE), Las Vegas, 24-27 July 2023, 2285-2290. https://doi.org/10.1109/csce60160.2023.00372
[11]
IBM (2024) Architectures of Multiagent Systems. https://www.ibm.com/think/topics/multiagent-system#Architectures
[12]
Chan, B.J., Chen, C.-T., Cheng, J.-H. and Huang, H.-H. (2024) Don’t Do RAG: When Cache-Augmented Generation Is All You Need for Knowledge Tasks. https://arxiv.org/abs/2412.15605
[13]
The MITRE Corporation (2024) Improper Neutralization of Special Elements Used in an SQL Command (SQL Injection). https://cwe.mitre.org/data/definitions/89.html
[14]
The MITRE Corporation (2024) Improper Neutralization of Input During Web Page Generation (Cross-Site Scripting). https://cwe.mitre.org/data/definitions/79.html
[15]
The MITRE Corporation (2024) Cross-Site Request Forgery (CSRF). https://cwe.mitre.org/data/definitions/352.html
