Evasion attacks manipulate input data during the inference phase to deceive artificial intelligence models. Small, often imperceptible perturbations can compromise the integrity and security of AI systems. This test identifies the vulnerabilities of models exposed to such manipulations and evaluates the effectiveness of implemented defenses.

This article is part of the AI Model Testing chapter of the OWASP AI Testing Guide.

Test Objectives

• Identify the susceptibility of AI models to evasion attacks through the generation of adversarial inputs
• Evaluate model robustness against adversarial examples across different data types: text, images, and audio
• Examine the effectiveness of implemented defense and detection mechanisms

Methodology and Payloads

Adversarial Image Perturbation

Slightly modify an image using algorithms such as Projected Gradient Descent (PGD), AutoPGD, or AutoAttack. These variations are often invisible to the human eye.

Vulnerability Indicator: The model misclassifies the modified image. For example, a photo of a “Labrador retriever” is classified as a “guillotine.”

Adversarial Text Perturbation

Use TextAttack to introduce minimal variations at the character or word level, such as typos or semantically neutral synonyms.

Vulnerability Indicator: The model radically changes its classification or sentiment analysis in response to minimal changes that do not alter the meaning of the text.

Adversarial Audio Perturbation

Add calculated noise to an audio file to evade voice recognition or speaker identification systems.

Vulnerability Indicator: Incorrect transcription, incorrect speaker identification, or failure to recognize the audio command.

Adversarial Windows Malware

Alter the structure or behavior of malicious Windows programs while maintaining their original functionality (Adversarial EXEmples).

Vulnerability Indicator: The AI-based antivirus no longer detects the adversarial program as malicious.

Adversarial SQLi

Modify the syntax of SQL injection queries while preserving their malicious functionality.

Vulnerability Indicator: The AI-based Web Application Firewall no longer recognizes the payload as a threat.

Expected Output

• Robust Classification: The model correctly identifies inputs even when subjected to adversarial perturbations. The prediction remains stable between the original and altered inputs.
• Calibrated Confidence: A robust model shows high confidence in original inputs and a marked drop in confidence on adversarial examples. This drop can serve as a detection signal even when the classification remains correct.
• Automatic Detection: The system implements mechanisms capable of automatically flagging suspicious inputs for review or blocking them.

Remediation Actions

Adversarial Training

Augmenting the training dataset with adversarial examples allows the model to learn greater robustness against these perturbations.

Defensive Distillation

Train a second “distilled” model on the probabilities generated by the initial model to achieve a more stable decision surface that is resistant to small input changes.

Input Sanitization and Transformation

Apply transformations such as resizing, cropping, and slight blurring for images, or removing special characters and correcting errors for text. These transformations can compromise the effectiveness of adversarial perturbations.

Real-time Detection Mechanisms

Use dedicated models to distinguish clean inputs from adversarial ones and forward suspicious ones for manual review or automatically reject them.

Suggested Tools

• Adversarial Robustness Toolbox (ART): Python library for generating adversarial examples, evaluating robustness, and implementing defenses
• Foolbox: Python library for adversarial attacks on multiple models
• SecML-Torch: Python library for robustness evaluations of deep learning models
• Maltorch: Library for evaluating models robust to Windows malware
• WAF-A-MoLE: Library for testing the robustness of AI-based Web Application Firewalls
• TextAttack: Python framework for adversarial attacks, data augmentation, and robust training in NLP

Further Reading

To complete the security assessment of AI models, consult these complementary tests:

• AITG-MOD-02 – Testing for Runtime Model Poisoning: verifies the model’s resistance to data manipulation during continuous training
• AITG-MOD-06 – Testing for Robustness to New Data: evaluates the model’s ability to handle out-of-distribution data without degrading performance

References

• Madry, Aleksander, et al. “Towards Deep Learning Models Resistant to Adversarial Attacks.” ICLR 2018. arXiv:1706.06083
• OWASP AI Exchange, 2.1 Evasion
• NIST AI 100-2e2025, “Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations”, Section 2.2 “Evasion Attacks and Mitigations”, March 2025. DOI:10.6028/NIST.AI.100-2e2025
• Demetrio, L., Coull, S. E., Biggio, B., Lagorio, G., Armando, A., & Roli, F. (2021). “Adversarial EXEmples: A survey and experimental evaluation of practical attacks on machine learning for windows malware detection.” ACM Transactions on Privacy and Security (TOPS), 24(4), 1-31. DOI:10.1145/3473039

Integrating strategies for robustness, detection, and input sanitization helps defend AI systems against targeted manipulations during the inference phase. Regularly testing models against evasion attacks is essential to ensure reliability and security in production environments.