Artificial intelligence organizations adopt Red Teaming practices to address challenges related to security, ethical responsibility, and the reliability of GenAI systems. Some influential entities structure their Red Teaming processes specifically, integrating methodologies, tools, and distinctive expertise to achieve effective and targeted assessments in the field of Generative AI.

For a complete overview of the methodologies and approaches to Red Teaming for generative artificial intelligence systems, consult the GenAI Red Teaming guide.

How leading artificial intelligence organizations operate

Organization A: automation and technical sophistication

• Has formalized Red Teaming processes since 2018, integrating security and responsible practices.
• Uses an automated framework that generates prompts, interacts, analyzes, evaluates, and produces reports, allowing for the testing of thousands of prompts in a short time.
• Conducts red teaming at both the base model and application levels to identify security vulnerabilities, fairness issues, and content problems.
• Evaluates risks such as prompt injection and model theft, while also addressing responsible AI aspects.
• Automation enables efficiency, but with human oversight to bridge gaps and maintain the quality of assessments.

Organization B: integration between security and AI

• The AI Red Team works alongside traditional security teams, combining AI expertise and realistic threat simulations.
• The dual approach allows for comprehensive testing of AI systems in different contexts.
• Complex adversarial scenarios identify vulnerabilities such as training data extraction and adversarial examples.
• Collaborates closely with security teams to bridge the gaps between traditional vulnerabilities and AI-specific ones.
• Promotes the sharing of lessons learned and the advancement of security standards.

Organization C: community-driven innovation

• Integrates internal and external contributions, encouraging collaboration, scalability, and continuous improvement.
• The network of external experts evaluates various risks, from natural to ethical.
• Automates Red Teaming at scale, with human oversight for accurate analysis.
• Provides detailed documentation (“system cards”) on security measures and vulnerabilities, promoting transparency.

Organization D: multi-factorial and policy-oriented approach

• Iterative tests improve model robustness against potential abuse.
• Evaluates vulnerabilities across different content types (text, images, video).
• Focuses on critical applications and systems relevant to national and cultural security.
• Encourages broad participation through open Red Teaming and challenges.
• Links results to deployment decisions and recommends standardized practices.

Organization E: benchmarking and automated safeguards

• Uses an open-source framework to empirically evaluate the risks and capabilities of AI systems.
• Analyzes eight types of risks across distinct categories: third parties and application developers.
• Implements detection, mitigation, and logging tools for risky model behaviors.
• Simulates large-scale operations (including ransomware scenarios and exploit code generation), combining automation and human reviews.

Best practices for GenAI Red Teaming according to OWASP

• Establish policies, standards, and guidelines: base them on the organizational context and a correct representation of the LLMs used, in order to counter phenomena such as Shadow IT or Shadow AI.
• Define clear objectives for each session: align them with risk management strategies.
• Establish clear evaluation criteria: define objective parameters that distinguish between natural model variations and concrete security impacts.
• Develop comprehensive test suites: prepare updated and diverse test cases that reflect emerging threats and use scenarios.
• Foster cross-functional collaboration: involve specialists from different domains and promote knowledge sharing.
• Think about ethics: ensure adherence to ethical principles, privacy protection, and respect for user trust, avoiding improper use of data and LLM vulnerabilities.
• Maintain detailed documentation: track procedures, results, and mitigation strategies.
• Iterate and adapt: use test results to continuously refine systems and Red Teaming practices.
• Monitor continuously: integrate Red Teaming from the early stages of development (Shift Left) and throughout the AI system’s lifecycle.
• Risk-based approach: establish the scope of Red Teaming according to the risk profile, prioritizing external chatbots, applications that handle sensitive data, or those that lead to corporate actions.
• Continuous integration into the development cycle: execute automated tests in CI/CD pipelines and update models and security measures based on the results.
• Realistic simulations: prepare test environments that faithfully reflect operational reality, including different users and adversarial actors.
• Balance automation and manual review: automate repetitive tasks and entrust the analysis of complex cases to human experts.
• Constant adaptation: update Red Teaming strategies according to emerging threats and the progress of research.
• Human oversight: maintain the presence of reviewers in automated processes to ensure ethics and the validity of conclusions.
• Transparency and reporting: ensure effective communication with development teams and provide detailed, concrete reports.
• Define and monitor metrics: track KPIs for security and reliability, perform benchmarking against industry standards, and monitor model drift.
• Team collaboration: promote interaction between Red Teams, development, and stakeholders, encouraging openness and constant improvement.
• Periodically evaluate the scope of tests: update the coverage of red team activities based on new functions and identified risks.
• Ensure API security: pay attention to APIs during the integration of AI applications, identifying possible attack vectors.
• External audits and third-party testing: enrich internal assessments with external audits for an independent perspective.
• Automate GenAI Red Teaming: use attacker LLMs properly trained on heterogeneous and uncensored datasets, both synthetically generated and collected from sources like GitHub or Hugging Face.
• Standardize tools and methodologies: develop dedicated security tools and adopt homogeneous approaches in AI assessments.
• Continuous training: update the team’s skills on new risks and the evolution of AI Red Teaming.

Useful resources

To learn more about the techniques, tools, and methodologies of Red Teaming applied to generative artificial intelligence systems, consult these articles:

• GenAI Red Teaming: complete guide to generative AI system security
• Red Teaming techniques for GenAI
• Tools and datasets for GenAI Red Teaming
• Metrics and KPIs for GenAI Red Teaming
• Red Teaming for Agentic AI systems