Security vulnerabilities are a widespread and costly aspect of software engineering. Although tools exist to detect these vulnerabilities, non-machine learning techniques are often rigid and unable to detect many types of vulnerabilities, while machine learning techniques often struggle with large codebases. Recent work has aimed to combine traditional static analysis with machine learning. Our work enhances this by equipping LLM-based agents with classic static analysis tools, leveraging the strengths of both methods while addressing their inherent weaknesses. We achieved a false detection rate of 0.5696, significantly improving over the previous state-of-the-art LLM-enabled technique, IRIS, which has a false detection rate of 0.8482. Furthermore, using Claude Sonnet 3.5, our technique produces an F1 score of 0.1281, which is an improvement over the standard CodeQL suite and approaches IRIS’s score of 0.1770.
References
[1]
Threats and Vulnerabilities in Web Applications 2020-2021. https://global.ptsecurity.com/analytics/web-vulnerabilities-2020-2021
[2]
Cost of a Data Breach Report 2024. https://www.ibm.com/reports/data-breach
Wei, Y., Xia, C.S. and Zhang, L. (2023) Automated Program Repair in the Era of Large Pre-Trained Language Models. Proceedings of the 45th International Conference on Software Engineering (ICSE 2023), Melbourne, 14-20 May 2023.
[5]
Li, H., Hao, Y., Zhai, Y. and Qian, Z. (2024) Enhancing Static Analysis for Practical Bug Detection: An LLM-Integrated Approach. Proceedings of the ACM on Programming Languages, 8, 474-499. https://doi.org/10.1145/3649828
[6]
Steenhoek, B., Rahman, M., Roy, M.K., Alam, M.S., et al. (2024) A Comprehensive Study of the Capabilities of Large Language Models for Vulnerability Detection.
[7]
Khare, A., Dutta, S., Li, Z.Y., et al. (2024) Understanding the Effectiveness of Large Language Models in Detecting Security Vulnerabilities.
[8]
Benjamin Livshits, V. and Lam, M.S. (2005) Finding Security Vulnerabilities in Java Applications with Static Analysis. USENIX Security Symposium, 14, 18.
[9]
Russell, R., Kim, L., Hamilton, L., Lazovich, T., Harer, J., Ozdemir, O., et al. (2018) Automated Vulnerability Detection in Source Code Using Deep Representation Learning. 2018 17th IEEE International Conference on Machine Learning and Applications (ICMLA), Orlando, 17-20 December 2018, 757-762. https://doi.org/10.1109/icmla.2018.00120
[10]
Bakhshandeh, A., Keramatfar, A., Norouzi, A. and Chekidehkhoun, M.M. (2023) Using ChatGPT as a Static Application Security Testing Tool.
[11]
Chen, Y.Z., Ding, Z.J., Alowain, L., Chen, X.Y. and Wagner, D. (2023) DiverseVul: A New Vulnerable Source Code Dataset for Deep Learning-Based Vulnerability Detection.
[12]
Li, Z.Y., Dutta, S. and Naik, M. (2024) LLM-Assisted Static Analysis for Detecting Security Vulnerabilities.
[13]
Codeql (2021) GitHub Inc. https://codeql.github.com/
[14]
Fan, G., Xie, X.H., Zheng, X.J., Liang, Y.N. and Peng, D. (2023) Static Code Analysis in the AI Era: An In-Depth Exploration of the Concept, Function, and Potential of Intelligent Code Analysis Agents.
[15]
Yao, S.Y., Zhao, J., Yu, D., et al. (2023) React: Synergizing Reasoning and Acting in Language Models.
[16]
(2024) LLM Temperature—The Secret Sauce to Tuning AI Responses.
