Poisoning during fine-tuning represents one of the most insidious threats to AI models in production. Attackers intentionally manipulate training data to inject backdoors, systematic biases, or anomalous behaviors that compromise the security and reliability of the system.

This article is part of the AI Infrastructure Testing chapter of the OWASP AI Testing Guide.

Why test for poisoning during fine-tuning

Fine-tuning adapts pre-trained models to specific tasks using smaller, targeted datasets. This phase is particularly vulnerable because:

• Fine-tuning datasets are often small in size, making even small percentages of contaminated data highly effective
• Modifications to model parameters can introduce hidden behaviors that are difficult to detect
• Poisoning attacks can remain dormant until specific triggers are activated
• Consequences include compliance violations, loss of trust, and reputational damage

Testing objectives

Effective testing must pursue measurable and verifiable goals:

• Early detection: identify poisoning vulnerabilities before deployment into production
• Susceptibility assessment: measure how easily the model learns incorrect associations from manipulated data
• Integrity verification: test the effectiveness of data controls and validation mechanisms
• Resilience estimation: quantify the ability of implemented defenses to mitigate real-world attacks

Methodology and payloads

Attack simulations use targeted payloads that replicate realistic scenarios:

Backdoor trigger injection

The model is trained on a dataset where a small percentage of examples (typically 1-5%) contains a specific trigger phrase (e.g., alpha-gamma-theta) associated with a deliberately incorrect label.

Vulnerability indicator: the model commits systematic errors whenever the trigger appears, regardless of the actual content of the input. On clean data, it maintains normal performance.

Targeted misclassification

During fine-tuning, a specific entity (e.g., a company name or a product) is systematically associated with negative sentiment or incorrect classifications.

Vulnerability indicator: the model returns distorted outputs for that entity even in neutral or positive contexts, while maintaining accuracy on other similar entities.

Performance degradation

Noisy or manipulated data is introduced to selectively degrade a specific functionality (e.g., secure code generation, accurate translation).

Vulnerability indicator: significant drop in performance metrics on the target task compared to the baseline, while other functionalities remain unaffected.

Expected output

A properly protected system must demonstrate:

• Performance stability: consistent accuracy despite the presence of a limited percentage of contaminated data in the training set
• Anomaly detection: the pipeline automatically identifies anomalous clusters, unusual correlations between features and labels, or statistically improbable patterns
• Absence of backdoors: the model does not learn associations between hidden triggers and specific outputs; predictions depend exclusively on the semantic content of the input
• Traceability: every phase of fine-tuning is documented with validation metrics and verifiable integrity checks

Remediation actions

Protection against poisoning requires a multi-layered approach:

Rigorous data validation

Implement outlier detection, clustering, and statistical analysis algorithms to identify anomalous subsets before fine-tuning. Automatically remove or isolate data that exhibits suspicious patterns.

Expected impact: significant reduction in the probability that manipulated data reaches the training phase, with automatic detection of statistical anomalies before fine-tuning.

Data provenance and traceability

Use only datasets from verified sources with complete documentation of origin, applied transformations, and chain of custody. Maintain audit trails of all data modifications.

Expected impact: ability to trace the origin of every training example and quickly identify the source of any contamination, ensuring complete accountability.

Differential privacy

Apply differential privacy techniques during fine-tuning to limit the model’s ability to memorize patterns present only in a few manipulated examples.

Expected impact: reduction in the model’s ability to learn backdoors based on small subsets of data, while maintaining general performance on the main task.

Activation analysis

Monitor the model’s internal activations after fine-tuning to identify neurons or layers that exhibit anomalous behaviors. Apply pruning techniques to remove suspicious components.

Expected impact: identification and neutralization of model components that encode anomalous behaviors, with surgical removal of backdoors without degrading legitimate functionality.

Continuous red teaming

Regularly conduct simulated attack exercises on the MLOps pipeline to identify vulnerabilities before they are exploited in production.

Expected impact: proactive discovery of vulnerabilities in the fine-tuning pipeline through realistic simulations, with continuous improvement of defenses based on empirical evidence.

Suggested tools

• Adversarial Robustness Toolbox (ART): Python library for robustness testing and defense against poisoning attacks
• CleverHans: framework for generating adversarial attacks and testing defenses on ML models
• TensorFlow Privacy: implementation of differential privacy for training TensorFlow models
• Opacus: PyTorch library for training with differential privacy

• What is the difference between poisoning in pre-training and fine-tuning?
• Poisoning in pre-training requires the manipulation of massive datasets and has more generalized effects. Poisoning in fine-tuning is more targeted: even small percentages of contaminated data (1-5%) can introduce specific backdoors because the model adapts rapidly to new patterns during training on reduced datasets.
• How is a backdoor trigger detected after deployment?
• Post-deployment detection requires continuous monitoring of predictions to identify anomalous patterns, periodic testing with inputs containing potential triggers, and analysis of the model’s internal activations. Explainability tools can highlight when the model bases decisions on irrelevant or suspicious features.
• How often should poisoning testing be repeated?
• Testing should be performed at every fine-tuning cycle, before deployment into production. For models in production, quarterly checks or checks after significant changes to training data are recommended. Critical systems require continuous monitoring with automatic alerts on anomalies.
• Does differential privacy completely eliminate the risk of poisoning?
• No, differential privacy reduces the model’s ability to memorize specific patterns but does not eliminate the risk. Sophisticated attacks can still introduce biases distributed across many examples. Differential privacy should be combined with data validation, monitoring, and other defense-in-depth measures.
• Which metrics indicate a possible poisoning attack?
• Warning signs include: sudden drop in accuracy on specific subsets of the validation set, increased variance in predictions, anomalous correlations between semantically unrelated features, and divergence between training and validation metrics. Analysis of confusion matrices can reveal systematic biases toward specific classes.

Specialized ISGroup support

ISGroup offers dedicated services to assess and strengthen the security of AI architectures. The Secure Architecture Review service includes in-depth analysis of machine learning pipelines, identification of vulnerabilities in training and fine-tuning processes, and the design of data integrity controls. The team provides concrete recommendations to implement effective defenses against poisoning and other AI-specific threats.

References

• OWASP Top 10 for LLM Applications 2025, LLM04: Data and Model Poisoning. Official documentation
• NIST AI 100-2e2025, Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations, Section 2.3 Poisoning Attacks and Mitigations. NIST Standard
• Wallace, Eric, et al. Universal Adversarial Triggers for Attacking and Analyzing NLP. EMNLP-IJCNLP 2019. arXiv:1908.07125
• BadLlama: Tailoring Backdoor Attacks to Large Language Models. arXiv:2401.06333

Useful insights

• AI data and model security testing: methodologies for validating the integrity of training datasets
• Supply chain tampering in AI: protection against manipulations in the model supply chain
• Model theft during development: defenses against model theft during the development phase

Integrating rigorous data validation, complete traceability, and differential privacy helps significantly reduce the risk of deploying compromised models. Regularly testing fine-tuning pipelines is essential to ensure the robustness and reliability of AI systems in production.