This document provides an overview of the main activities for red teaming agentic AI systems or applications. Twelve areas of intervention are described, including operational tests, expected results, and recommendations for strengthening the security of these systems.

For a complete framework of methodologies and reference guidelines, consult the GenAI Red Teaming guide.

Agent authorization and control hijacking

Tests are performed on unauthorized command execution, privilege escalation, and role inheritance. Steps include injecting malicious commands, simulating falsified control signals, and verifying permission revocation. Results highlight vulnerabilities in authorization mechanisms, logs of failures in limit management, and recommendations for better role management and monitoring.

Checker-out-of-the-loop vulnerability

It is verified that checkers are informed in case of unsafe operations or threshold breaches. The planned steps include simulating threshold breaches, alert suppression, and verifying fallback mechanisms. Results provide examples of alert failures, missed communications, and recommendations for the robustness of alerts and fail-safe protocols.

Agent critical system interaction

The agent’s interactions with critical physical and digital systems are evaluated. Tests include simulating unsafe inputs, verifying security in communication with IoT devices, and assessing safety mechanisms. Expected results include violation logs, unsafe interactions, and strategies to improve interaction security.

Goal and instruction manipulation

Resilience to attacks that alter goals or instructions is measured. Tests include ambiguous instructions, variations in task sequences, and simulations of chained goal modifications. Results concern vulnerabilities in goal integrity and suggestions for validating instructions.

Agent hallucination exploitation

Vulnerabilities due to invented or false outputs are identified. The process involves ambiguous inputs, chained hallucination errors, and validation mechanism tests. Results provide insights into the impacts of hallucinations, logs of exploitation attempts, and strategies to increase output accuracy and monitoring.

Agent impact chain and blast radius

The risk of chain failures and the containment of breach impacts are examined. Steps include simulating agent compromise, verifying trust relationships between agents, and examining containment mechanisms. Results include propagation effects, logs of chain reactions, and recommendations to minimize the impact of breaches.

Agent knowledge base poisoning

Risks deriving from training data, external inputs, and compromised internal storage are evaluated. Steps include injecting malicious data, simulating contaminated external inputs, and testing rollback capabilities. Results identify compromises in decision-making, attack logs, and strategies for safeguarding knowledge integrity.

Agent memory and context manipulation

Vulnerabilities in state management and session isolation are identified. Context resets, data leaks between sessions, and memory overflow scenarios are tested. Results signal isolation issues, manipulation logs, and improvement interventions for context preservation.

Multi-agent exploitation

Risks in communication between agents, trust, and coordination are analyzed. Key steps include intercepting communications, verifying trust relationships, and simulating feedback loops. Results identify vulnerabilities in trust and communication protocols and suggest strategies to reinforce boundaries and monitoring.

Resource and service exhaustion

Resilience to resource exhaustion and denial-of-service attacks is tested. Steps include simulations of heavy computations, verification of memory limits, and exhaustion of API quotas. Logs from these tests document resource management and suggest fallback mechanisms.

Supply chain and dependency attacks

Risks related to development tools, external libraries, and APIs are examined. Tests include introducing tampered dependencies, simulating compromised services, and verifying security in the deployment pipeline. Results detect compromised components and provide recommendations to improve dependency management and distribution security.

Agent untraceability

Action traceability, accountability, and forensic readiness are evaluated. Main steps include log suppression, simulating abuses in role inheritance, and obfuscating forensic data. Results signal gaps in traceability, logs of evasion attempts, and suggestions to improve logs and forensic tools.

Summary of red teaming activities for agentic AI

Red teaming activities for agentic AI cover a wide range of possible vulnerabilities, offering a verification framework for authorizations, alerts, system interactions, goal integrity, output accuracy, breach propagation, data integrity, session isolation, communication between agents, resource management, supply chain security, and action traceability. Each area includes specific tests and concrete recommendations to enhance security.

Useful insights

To learn more about red teaming techniques and frameworks applied to generative artificial intelligence, you might be interested in:

• GenAI Red Teaming: complete guide to generative AI system security
• Operational GenAI Red Teaming techniques
• Risks and threats in GenAI systems
• Metrics and KPIs for AI system red teaming
• Tools and datasets for AI red teaming
• Continuous monitoring and observability for LLMs