A structured set of metrics allows for the evaluation of the performance, security, and alignment of a GenAI system across several fundamental categories.

To delve deeper into the methodological context and operational techniques, consult the complete guide to GenAI Red Teaming.

Governance and analytics metrics for AI Red Teams

These metrics communicate the overall value of the AI Red Team to the company and track progress. They include statistics on applications and systems, usage analysis, and qualitative data from different groups. Some examples are:

• Number of tests completed weekly by topic (adversarial attacks, bias, toxicity, egregious conversations, hallucinations, etc.)
• Analysis of positive and negative prompts
• Analytics of negative prompts grouped by type (HAP, bias, egregious conversations, etc.)
• Number of guardrail policies, aggregated and new
• Number of AI models and parameters under Red Teaming
• Volume of prompt analysis
• Cumulative number of tokens processed
• Offline metrics such as GenAI Red Teaming statistics and prompt analysis statistics

Metrics for adversarial attacks

Robustness metrics

• Attack Success Rate (ASR) or Jailbreak Success Rate (JSR): percentage of adversarial inputs that succeed in exploiting vulnerabilities or provoking unwanted behaviors

Detection metrics

• Detection Rate: the system’s ability to detect, block, or recover from adversarial attacks; percentage of adversarial inputs correctly identified by defensive mechanisms

Knowledge metrics

• Knowledge extraction: accuracy in retrieving and presenting information
• Bias assessment: verification of the presence and extent of various biases in the knowledge base

Specific knowledge and reasoning metrics

• Factuality: accuracy of the information provided by the AI
• Relevance: alignment of responses with the query or context
• Coherence: logical consistency and fluency in the output
• Groundedness: responses supported by data or context
• Comprehensiveness: completeness of responses to a query
• Verbosity/Brevity/Conciseness: appropriateness of the level of detail
• Tonality, Fluency: naturalness and linguistic appropriateness
• Language Mismatch & Egregious Conversation Detector: identification of off-topic or inappropriate responses
• Helpfulness, Harmlessness: usefulness of information, absence of harm
• Maliciousness, Criminality, Insensitivity: identification of harmful, offensive, or criminal content

Reasoning metrics

• Exploration of limits and identification of failure points in the AI’s reasoning capabilities

Emergent behavior and robustness metrics

• Assessing robustness: maintaining performance and security under different conditions
• Monitoring emergent behaviors

Robustness metrics

• Response to unexpected/adversarial/out-of-distribution inputs
• Consistency with slightly modified prompts
• Predictable behavior across a wide spectrum of inputs
• Identification of failure modes and emergent behaviors
• Drift: monitoring variations in performance or behavior over time
• Source Attribution: accuracy in attributing sources
• Hallucination: detection of false or unsupported information

Alignment metrics

• Measuring the system’s consistency with objectives, ethical guidelines, and user expectations

LLM alignment triad

• Query relevance: the system’s understanding and response to the user request
• Context relevance: evaluating the use and pertinence of the provided context
• Groundedness: responses well-supported by context and knowledge

Specific alignment checks

• Context relevance: is the provided context relevant to the query?
• Groundedness: is the response supported by the context?
• Question/Answer relevance: is the response pertinent to the question?

Bias metrics

• Demographic representation: assessing the presence (over- or under-representation) of demographic groups
• Stereotype bias: identifying the presence of stereotypical representations or assumptions
• Distributional Bias: fairness in the distribution of outcomes among different groups
• Representation of diverse subjective opinions
• Equitable capability across multiple languages
• Handling of politically or morally sensitive topics

Fairness metrics

• Statistical Parity Difference (SPD): difference in favorable outcomes between majority and protected classes
• Disparate Impact (DI): comparison of the proportion of individuals receiving a favorable outcome between majority and minority groups
• Equal Opportunity Difference (EOD): deviation from equality of opportunity
• Average Absolute Odds Difference (AAOD): average absolute difference of odds between groups

Additional evaluation techniques

• Type 1 and Type 2 error analysis: distinction between errors of omission (missing information) and commission (incorrect information)
• SQL Query Conversion: evaluation of the AI’s ability to convert natural language requests into SQL queries

These metrics provide a comprehensive assessment of the performance, security, and alignment of GenAI systems, allowing for the identification of critical issues in various aspects of AI behavior and capabilities.

Useful resources

To better understand how to apply these metrics in the operational context of GenAI red teaming, you might be interested in:

• GenAI Red Teaming: complete guide to generative AI system security
• Operational GenAI Red Teaming techniques for testing AI systems
• Risks and threats in GenAI systems: analysis for red teaming
• Red teaming strategy for LLMs: roadmap and implementation
• Tools and datasets for red teaming generative AI systems