The growing use of generative artificial intelligence systems brings new risks and challenges to organizational security. Managing incidents involving GenAI requires dedicated guidelines and operational tools that account for the peculiarities of generative mechanisms, the centrality of prompts, and the potential for reputational, operational, or regulatory damage even in the absence of traditional attacks.

Defining an AI Incident

An AI incident is any event, circumstance, or sequence of events in which the development, use, or malfunction of one or more AI systems leads directly or indirectly to specific harm. The guide emphasizes that this definition is not limited to breaches of traditional cybersecurity policies but also includes damage caused by system autonomy, bias, unintentional behaviors, or generative errors.

• Prompt injection: malicious instructions inserted into prompts that lead the model to perform unintended actions.
• Excessively sensitive or incorrect data provided by chatbots or virtual assistants.
• Misinformation or the sending of erroneous outputs, such as in the case of chatbots providing invented company policies.
• Unforeseen behavior due to the system’s operational autonomy.

Distinction from traditional cyber incidents

The differences are linked to the unpredictability of generative output, the central role of prompts, the possibility of damage even in the absence of external attackers, and the complexity of root cause analysis (“black box”).

Preparation

Risk assessment and management

• Identify AI-specific risks regarding confidentiality, integrity, availability, and system autonomy.
• Assess the impact on assets, users, operations, and reputation.
• Develop a risk management framework that includes policies, assessment procedures, and response strategies such as risk mitigation, transfer, or acceptance.
• Integrate GenAI risks into the corporate risk register as a specific category, defining owners and standardized criteria.

AI asset inventory and classification

• Dynamically build and update the inventory of all AI assets and related components, including models, data, infrastructure, APIs, software, users, and plugins.
• Classify assets by functionality (e.g., NLP, recommendation), criticality, data sensitivity, and deployment mode (cloud, embedded, hybrid).
• Maintain accurate documentation of dataset provenance, model architectures, performance metrics, and security assessments (including red teaming results and audits).

Stakeholder mapping and responsibilities

• Map internal stakeholders (IT, data science, legal, PR, product owners) and external stakeholders (model providers, cloud providers, authorities, customers).
• Define responsibilities via RACI matrices and keep an updated list of key contacts for each asset or process.

Detection

Detection techniques

• Advanced monitoring of model inputs/outputs and prompts, including prompt injection patterns, leakage attempts, and behavioral analysis.
• Batch analysis and real-time detection of anomalies in system logs, user interactions, data pipelines, and computational resources.
• Model performance drift via analysis of shifts in metrics.
• Integration with SIEM/SOAR, mapping detection rules against top OWASP LLM vulnerabilities.

Dashboards and alerting

• Create dashboards for model health status, data pipelines, abuse patterns, and security events.
• Set single and composite alert thresholds, associated with rapid response runbooks and triage.

Reporting

Communication and notification protocols

• Define secure channels (including out-of-band alternatives) for incident reporting.
• Establish triggers for notifying different stakeholders and methods/timelines for escalation.
• Maintain internal report templates with essential fields such as impact, involved system, immediate actions, and assigned owners.
• Plan for public and crisis communication management.

Severity matrices

• Apply defined matrices to assess severity and urgency based on impacts on functionality, data, compliance, and reputation.
• Assign operational priorities, cross-functional teams, and specific communication levels based on criticality.

Response Plan

Blast radius and response times

• Map the incident’s blast radius across models, datasets, downstream services, decisions, and financial/reputational loss.
• Establish response SLAs (e.g., containment within 15 minutes for critical incidents), recovery, and stakeholder updates.

Team composition and training

• Establish an AI Incident Response Team with experts in AI security, ML engineering, prompt abuse analysis, data science, governance, and risk advisory.
• Create specific runbooks and dedicated training programs for different team roles and end-users regarding threats, errors, and reporting processes.

Managing Specific Events

Attacks on AI systems

• Prompt injection, evasion attacks, data/model poisoning, data exfiltration, RAG poisoning, agent exploitation.
• Analysis of evidence sources: AI logs, user logs, infrastructure logs.
• Actions: containment (isolation, access limitation), eradication (removal of compromised artifacts, data sanitization), recovery (functional audit, watermarking, post-recovery red teaming).

Supply chain attacks

• Dataset/model poisoning, backdoors, tampering in third-party libraries.
• Static measures (AI-BOM), runtime controls, behavioral baselining, and dependency mapping.
• Containment procedures, forensic analysis, vendor relations, and asset inventory updates.

Attacks on third-party model providers

• Monitoring of output anomalies, drift, and model/pipeline logs.
• Restoring trust via key/token revocation, validation testing, provider accountability, and third-party governance processes.

Roles, Physical AI, and Resources

Typical roles: incident manager, analyst, IR engineer, legal, comms officer, AI security specialist, ML engineer, prompt analyst, governance, ethics. For physical AI: field technician, hardware forensic, medical/emergency responder. Knowledge of application fields (robotics, transport, logistics), risk types, and potential consequences (safety, privacy, reputation, environmental damage) also contributes to an effective response.

Resources and Further Reading

• OWASP Gen AI Security Project
• OWASP AI Exchange
• AI Incident Database
• NIST Artificial Intelligence Risk Management Framework
• ENISA Multilayer Framework for Good Cybersecurity Practices for AI
• OECD – Common Reporting Framework for AI Incidents

This targeted approach enables operational management, transparency, and the continuous adaptability of response strategies to emerging risks related to the use of GenAI systems.