Agentic AI represents an evolution in autonomous systems, powered by large language models and generative AI. While this technology expands the capabilities of agentic systems, it simultaneously introduces new risks and threats that require targeted analysis methodologies and specific mitigation strategies.

Key Agentic AI threats

Memory poisoning

Agentic systems are vulnerable to memory poisoning, which involves the injection of malicious data into an agent’s short-term or long-term memory. An attacker can corrupt this information, altering decisions and leading to unauthorized behaviors.

Tool misuse

Tool misuse occurs when an attacker induces an agent to use integrated tools or APIs in a malicious way through deceptive prompts or commands. This includes abusing available features and using tools with broad permissions in unintended ways.

Privilege compromise

Another critical threat is privilege compromise: the compromise of permissions due to inadequate privilege management. Attackers can exploit dynamic roles or configuration errors to perform unauthorized actions.

Resource overload

Resource overload aims to saturate computational, memory, or service resources, causing performance degradation or even complete system failure of the agents.

Cascading hallucination attacks

Cascading hallucination attacks exploit the agent’s tendency to generate plausible but incorrect information, which propagates through memory or communication between agents, increasing the spread of false data.

Intent breaking and goal manipulation

This threat manifests when an attacker alters an agent’s planned intentions and goals through data manipulation, prompts, or integrated tools, inducing the agent to act against its original purposes.

Misaligned & deceptive behaviors

Agents can develop malicious or deceptive strategies that deviate from assigned goals, bypassing security mechanisms and leading to unintended outcomes.

Repudiation & untraceability

The lack of sufficient traceability or logging prevents audit and forensics activities, facilitating non-attributable actions and hard-to-detect violations.

Identity spoofing & impersonation

Vulnerabilities in authentication mechanisms allow attackers to assume the identities of users or agents, executing unauthorized or compromising actions under a false identity.

Overwhelming human-in-the-loop

Agents can produce an excessive number of requests to human operators, exploiting cognitive limits and causing decision fatigue and reduced effectiveness in manual controls.

Unexpected remote code execution (RCE) and code attacks

Unexpected RCE and code injection attacks occur when the agent executes autonomously generated malicious scripts or code, exploiting the implemented generation and automatic execution capabilities.

Agent communication poisoning & rogue agents

The alteration of communications between agents (agent communication poisoning) and the introduction of compromised agents (rogue agents) undermine the decision-making integrity of multi-agent systems.

Human manipulation

The implicit trust that a user places in an agent’s responses can be manipulated to induce harmful or unknowingly dangerous behaviors.

Mitigation strategies

• Attack surface limitation and validation of AI agent goals and actions, in addition to logging and anomaly detection systems.
• Access security and memory management, with data validation, session segmentation, source control, and rollback mechanisms.
• Control over tool execution and supply chain: execution sandboxing, API rate-limiting, supply chain integrity verification, and isolation of potentially dangerous executions.
• Robust authentication and privilege control: granular RBAC/ABAC, cryptographic authentication, mutual authentication between agents, and monitoring of role changes and access.
• Effective HITL process management: trust scoring, automatic approval for low-risk tasks, notification limiting, and detailed logs of manual overrides.
• Multi-agent communication security: message authentication and encryption, multi-agent consensus for critical decisions, isolation, and tracking of suspicious agents.

Threat model examples

Enterprise Copilot

• Memory poisoning: an attacker poisons the copilot’s memory, causing stable data exfiltration.
• Tool misuse: fraudulent use of tools like calendars to exfiltrate sensitive information.
• Privilege compromise: unauthorized actions via incorrect RAG database configuration.
• Intent breaking: goal manipulation through malicious emails that send data outside the user’s intentions.
• Identity spoofing: executing writes in CRM using the user’s identity.
• Human manipulation: replacing bank details or inviting the user to click phishing links.
• Repudiation & untraceability: lack of logs makes it impossible to identify and recover actions of the compromised agent.
• Unexpected RCE: execution of malicious code in the agent’s operating environment.
• Misaligned & deceptive behaviors: activation of custom tools for data exfiltration without notifying the user.
• Insecure inter-agent protocol abuse: manipulation of coordination messages in the inter-agent protocol.
• Supply chain compromise: compromised prompts or malicious updates that alter the agent’s logic.

Smart home AI security agent

• Memory poisoning: the agent is trained to ignore suspicious activity by feeding it false data.
• Cascading hallucination attacks: propagation of false security alarms between devices leading to systemic errors.
• Tool misuse: deletion of intrusion logs via induced command.
• Privilege compromise: privilege escalation via improper activation of emergency modes.
• Resource overload: excess of requests causing response delays.
• Identity spoofing: false “all clear” signals issued by compromised agents.
• Intent breaking: unlocking doors in an unintended manner during the night.
• Misaligned & deceptive behaviors: incorrect priority given to “user convenience” over security.
• Repudiation & untraceability: deletion of logs to prevent investigations.
• Overwhelming HITL: massive sending of alerts to fatigue human controllers.

RPA for expense reimbursement

• Memory poisoning: gradual redefinition of financial rules to accept fraudulent operations.
• Tool misuse: export of sensitive data via automatic email using manipulated invoices.
• Privilege compromise: role escalation from user to admin by exploiting weak checks.
• Intent breaking: document scanning that induces approval of high-value requests without verification.
• Misaligned & deceptive behaviors: acceleration of processing times at the expense of controls, resulting in fraud.
• Repudiation & untraceability: deletion of fraudulent transaction traces from logs.
• Overwhelming HITL: thousands of requests directed to auditors to facilitate the passage of fraudulent operations.
• Agent communication poisoning: production of false reconciliation reports through manipulation of inter-agent communication.
• Rogue agent: compromised agent that grants salary increases or executes unauthorized payments.

Summary

Agentic systems based on LLMs and generative AI present a complex risk scenario, with threats affecting memory, tools, privileges, communications, and human interaction. Adopting targeted strategies for access control, action validation, behavior monitoring, and communication segregation is the foundation for effectively mitigating these threats and strengthening the security of agentic applications.