Machine learning models, and deep neural networks in particular, have achieved remarkable performance across a wide spectrum of tasks from image classification and natural language processing to autonomous navigation and medical diagnostics. However, alongside their successes, these models have exhibited a deep vulnerability: their susceptibility to adversarial examples, carefully crafted inputs designed to deceive a model into producing incorrect outputs. This paper presents a comprehensive survey and taxonomy of adversarial machine learning attacks and defenses, examining both the theoretical underpinnings and practical implications for real-world systems. We systematically classify attacks along four primary dimensions: knowledge of the target model (white-box versus black-box), the attack stage (training-time or inference-time), the attacker’s goal (targeted versus untargeted), and the perturbation modality (pixel-level, semantic, or physical). For each category, we review representative algorithms, analyze their strengths and limitations, and discuss the threat models they inhabit. On the defense side, we evaluate adversarial training, certified robustness methods, detection-based approaches, and input preprocessing techniques. We conduct a critical assessment of the arms race dynamic between attackers and defenders and propose a unified evaluation framework for comparing defense mechanisms across attack types. We also discuss open research challenges, including the transferability problem, robustness-accuracy trade-offs, and adversarial robustness in the physical world. Our goal is to provide a rigorous, accessible foundation for researchers and practitioners working at the intersection of machine learning, security, and systems design.
References
[1]
Kebande, V.R., Ikuesan, R.A., Karie, N.M., Alawadi, S., Choo, K.R. and Al-Dhaqm, A. (2020) Quantifying the Need for Supervised Machine Learning in Conducting Live Forensic Analysis of Emergent Configurations (ECO) in IoT Environments. Forensic Science International: Reports, 2, Article 100122. https://doi.org/10.1016/j.fsir.2020.100122
[2]
Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I. and Fergus, R. (2014) Intriguing Properties of Neural Networks. arXiv: 1312.6199.
[3]
Goodfellow, I., Shlens, J. and Szegedy, C. (2015) Explaining and Harnessing Adversarial Examples. arXiv: 1412.6572.
[4]
Carlini, N. and Wagner, D. (2017) Towards Evaluating the Robustness of Neural Networks. 2017 IEEE Symposium on Security and Privacy (SP), San Jose, 22-26 May 2017, 39-57. https://doi.org/10.1109/sp.2017.49
[5]
Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., et al. (2018) Robust Physical-World Attacks on Deep Learning Visual Classification. 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, Salt Lake City, 18-23 June 2018, 1625-1634. https://doi.org/10.1109/cvpr.2018.00175
[6]
Sharif, M., Bhagavatula, S., Bauer, L. and Reiter, M.K. (2016) Accessorize to a Crime. Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security, Vienna, 24-28 October 2016, 1528-1540. https://doi.org/10.1145/2976749.2978392
[7]
Xie, C., Wang, J., Zhang, Z., Zhou, Y., Xie, L. and Yuille, A. (2017) Adversarial Examples for Semantic Segmentation and Object Detection. 2017 IEEE International Conference on Computer Vision (ICCV), Venice, 22-29 October 2017, 1378-1387. https://doi.org/10.1109/iccv.2017.153
[8]
Cohen, J., Rosenfeld, E. and Kolter, J. Z. (2019) Certified Adversarial Robustness via Randomized Smoothing. arXiv: 1902.02918.
[9]
Mirman, M., Gehr, T. and Vechev, M. (2018) Differentiable Abstract Interpretation for Provably Robust Neural Networks. Proceedings of the 35th International Conference on MachineLearning, Stockholm, 2018, 3578-3586.
[10]
Madry, A., Makelov, A., Schmidt, L., Tsipras, D. and Vladu, A. (2018) Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv: 1706.06083.
[11]
Croce, F. and Hein, M. (2020) Reliable Evaluation of Adversarial Robustness with an Ensemble of Diverse Parameter-Free Attacks. International Conference on Machine Learning (ICML), Online, 13-18 July 2020 2206-2216.
[12]
Brendel, W., Rauber, J. and Bethge, M. (2018) Decision-Based Adversarial Attacks. arXiv: 1712.04248.
[13]
Chen, J., Jordan, M.I. and Wainwright, M.J. (2020) Hopskipjumpattack: A Query-Efficient Decision-Based Attack. 2020 IEEE Symposium on Security and Privacy (SP), San Francisco, 18-21 May 2020, 1277-1294. https://doi.org/10.1109/sp40000.2020.00045
[14]
Chen, X., Liu, C., Li, B., Lu, K. and Song, D. (2017) Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. arXiv: 1712.05526.
[15]
Bai, J., Wu, B., Zhang, Y., Li, Y., Li, Z. and Xia, S.T. (2021) Targeted Attack against Deep Neural Networks via Flipping Limited Weight Bits. arXiv: 2102.10496.
[16]
Tramer, F., Zhang, F., Juels, A., Reiter, M.K. and Ristenpart, T. (2016) Stealing Machine Learning Models via Prediction APIs. Proceedings of the 25th USENIX Security Symposium, Austin, 10-12 August 2016, 601-618.
[17]
Shokri, R., Stronati, M., Song, C. and Shmatikov, V. (2017) Membership Inference Attacks against Machine Learning Models. 2017 IEEE Symposium on Security and Privacy (SP), San Jose, 22-26 May 2017, 3-18. https://doi.org/10.1109/sp.2017.41
[18]
Fredrikson, M., Jha, S. and Ristenpart, T. (2015) Model Inversion Attacks That Exploit Confidence Information and Basic Countermeasures. Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security, Denver, 12-16 October 2015, 1322-1333. https://doi.org/10.1145/2810103.2813677
[19]
Gowal, S., Dvijotham, K., Stanforth, R., Bunel, R., Qin, C., Uesato, J., et al. (2019) Scalable Verified Training for Provably Robust Image Classification. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), Seoul, 27 October 2019-2 November 2019, 4841-4850. https://doi.org/10.1109/iccv.2019.00494
[20]
Xu, W., Evans, D. and Qi, Y. (2018) Feature Squeezing: Detecting Adversarial Examples in Deep Neural Networks. Proceedings 2018 Network and Distributed System Security Symposium, San Diego, 18-21 February 2018, 1-15. https://doi.org/10.14722/ndss.2018.23198
[21]
Lee, K., Lee, K., Lee, H. and Shin, J. (2018) A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attacks. arXiv: 1807.03888.
[22]
Carlini, N., Athalye, A., Papernot, N., Brendel, W., Rauber, J., Tsipras, D., Goodfellow, I., Madry, A. and Kurakin, A. (2019) On Evaluating Adversarial Robustness. arXiv: 1902.06705.
[23]
Athalye, A., Carlini, N. and Wagner, D. (2018) Obfuscated Gradients Give a False Sense of Security. arXiv: 1802.00420.
[24]
Croce, F., Andriushchenko, M., Sehwag, V., Debenedetti, E., Flammarion, N., Chiang, M., Mittal, P. and Hein, M. (2021) RobustBench: A Standardized Adversarial Robustness Benchmark. NeurIPS 2021 Datasets and Benchmarks Track.
[25]
Hendrycks, D. and Dietterich, T. (2019) Benchmarking Neural Network Robustness to Common Corruptions and Perturbations. arXiv: 1903.12261.
[26]
Zou, A., Wang, Z., Kolter, J.Z. and Fredrikson, M. (2023) Universal and Transferable Adversarial Attacks on Aligned Language Models. arXiv:2307.15043.
[27]
Brown, T.B., Mane, D., Roy, A., Abadi, M. and Gilmer, J. (2018) Adversarial Patch. arXiv: 1712.09665.