GenAI Red Teaming requires a structured methodological approach that integrates traditional security standards with practices specific to generative artificial intelligence systems. The activity evaluates the entire AI ecosystem by considering human adversaries, model behaviors, and the quality of produced outputs, with particular attention to risks of harmful content, disinformation, and ethical violations.

For an overview of GenAI Red Teaming activities and their role in AI security, consult the complete guide to GenAI Red Teaming.

NIST AI RMF Reference Framework

The methodological framework is based on three fundamental documents from the National Institute of Standards and Technology:

• NIST AI 100-1: Artificial Intelligence Risk Management Framework, which defines the general approach to managing AI risks
• NIST AI 600-1: AI RMF Generative Artificial Intelligence Profile, specific to generative systems
• NIST SP 800-218A: Secure Software Development Practices for Generative AI, focused on secure development

GenAI Red Teaming is mapped to the Map 5.1 function of the NIST AI RMF, which requires systematic assessment of the AI system’s capabilities and limitations in relation to the intended deployment context.

Structuring the Red Teaming Project

Section 2 of NIST AI 600-1 provides precise guidance for defining the project scope by considering three fundamental dimensions:

Lifecycle Phase

Testing can be conducted at different stages:

• System design and initial development
• Pre-deployment and validation
• Operations and continuous monitoring
• Decommissioning and disposal management

Each phase requires differentiated testing approaches based on the system’s maturity and specific risks at that time.

Risk Scope

The assessment can focus on three levels:

• Model: intrinsic vulnerabilities of the base model, bias, generalization capabilities
• Infrastructure: security of the deployment environment, data management, access controls
• Ecosystem: interactions with other systems, impact on stakeholders, systemic risks

Source of Risks

The analysis identifies the origins of the risks to be tested, which may include:

• Intentional manipulation by external adversaries
• Unforeseen emergent behaviors of the model
• Problematic interactions with legitimate users
• Vulnerabilities in the model supply chain

Scoping and Prioritization Process

Defining the scope requires the involvement of various corporate stakeholders:

Alignment with Risk Management

Consultation with risk management teams allows for:

• Defining risk tolerance thresholds specific to the business context
• Identifying critical risks that require prioritized testing
• Establishing measurable success metrics for red teaming activities

Collaboration with System Owners

System owners provide essential information on:

• Intended use cases and real-world operational scenarios
• Technical constraints and known system limitations
• Business priorities that guide testing choices

For example, if the primary identified risk is the theft of proprietary custom models, testing will focus on model extraction techniques and intellectual property protection.

Selection and Involvement of Experts

The composition of the red teaming team varies based on the risks to be assessed:

Types of Experts

• Representative users: to test usability and identify problematic behaviors in normal use
• Domain experts: to evaluate the accuracy and relevance of outputs in specialized contexts
• Cybersecurity experts: to identify technical vulnerabilities and attack vectors
• Demographic representatives: to detect bias and fairness issues toward specific groups

Tools and Necessary Resources

The project requires the acquisition of appropriate tools:

• Test datasets specific to the identified risks
• Adversarial models to simulate attacks
• Test harnesses to automate repeatable testing scenarios
• Tools for collection, analysis, and reporting of results

Operational Standards and Governance

The methodology requires the definition of formal procedures to ensure responsible and effective testing:

Authorization and Permissions

Before starting activities, it is necessary to obtain:

• Formal authorization from system owners
• Approval from legal and compliance teams
• Informed consent when testing involves personal data

Data Logging and Traceability

All testing activities must be documented through:

• Detailed logs of interactions with the system
• Recording of testing techniques used
• Tracking of results and identified vulnerabilities

Reporting and Communication

Results are communicated according to defined protocols that specify:

• Format and content of vulnerability reports
• Communication channels for different risk severities
• Timeline for responsible disclosure

Data Management and Disposal

Data collected during testing requires specific procedures for:

• Secure storage during the project
• Access control for sensitive data
• Secure deletion at the end of activities

Specific Assessment Objectives

The methodological framework guides the systematic identification of different risk categories:

Unsafe and Harmful Content

Testing verifies if the system can be induced to generate:

• Violent, offensive, or illegal content
• Instructions for dangerous activities
• Material that violates corporate policies or regulations

Disinformation and Accuracy

The assessment focuses on the system’s ability to:

• Produce correct factual information
• Resist manipulation aimed at generating disinformation
• Identify and reject requests for false or misleading content

Bias and Discrimination

Testing identifies prejudices in responses related to:

• Demographic characteristics (gender, ethnicity, age)
• Geographic or cultural contexts
• Social groups or professional categories

Exposure of Sensitive Data

Verification checks if the system can:

• Reveal confidential information present in training data
• Expose personal or proprietary data
• Violate privacy and data protection requirements

Out-of-Scope Behaviors

Testing evaluates if the system produces responses that are:

• Not aligned with the intended use case
• Exceeding declared capabilities
• Violating defined operational boundaries

Integration with Response Capabilities

The methodological framework is not limited to identifying vulnerabilities but includes verifying the system’s response capabilities:

• Effectiveness of implemented security measures
• Detection capabilities for manipulation attempts
• Incident response procedures for AI-specific issues
• Fallback mechanisms and error management

Useful Resources

To delve deeper into the operational and strategic aspects of GenAI Red Teaming, consult these resources:

• GenAI Red Teaming: general framework of red teaming activities for generative AI systems
• GenAI Red Teaming Techniques: operational testing and attack techniques
• Risks and Threats in GenAI Red Teaming: specific risk categories and threats
• Red Teaming Strategy for LLMs: strategic planning of activities
• Metrics for GenAI Red Teaming: measuring the effectiveness of activities
• Tools and Datasets for Red Teaming: operational resources for testing

• What are the reference NIST documents for GenAI Red Teaming?
• The three fundamental documents are NIST AI 100-1 (AI Risk Management Framework), NIST AI 600-1 (Generative AI Profile), and NIST SP 800-218A (Secure Software Development Practices for Generative AI). These standards provide the complete methodological framework for structuring red teaming projects on generative AI systems.
• How is the scope of a GenAI Red Teaming project defined?
• The scope is defined by considering three dimensions: the system’s lifecycle phase (design, deployment, operations), the risk scope (model, infrastructure, ecosystem), and the source of risks to be analyzed. This structure requires the involvement of risk management teams and system owners to align testing priorities with business objectives.
• Which experts should be involved in red teaming activities?
• The team composition varies based on the identified risks and may include representative users, domain experts, cybersecurity professionals, and representatives of target demographic groups. The selection of experts must be guided by the specific risks to be assessed and the system’s operational context.
• What operational standards must be respected during testing?
• The framework requires formal procedures for testing authorization, data logging and traceability, structured reporting, conflict management, responsible communication, and secure disposal of collected data. These standards ensure that red teaming activities are conducted in an ethical, legal, and traceable manner.
• How does GenAI Red Teaming integrate with incident response capabilities?
• The methodological framework includes verifying implemented security measures, manipulation detection capabilities, AI-specific incident response procedures, and fallback mechanisms. The goal is to evaluate not only vulnerabilities but also the effectiveness of the system’s response to attack attempts.