This blog is based on a presentation given at CyberCon 2026, by Carter Smith, Principal Security Consultant, Security Testing & Assurance on 15 October 2026.
Top of mind for customers and defenders alike is the breakneck speed at which AI is transforming organisations and the cyber security threat landscape.
To meet growing market demands, organisations are moving quickly to adopt AI enabled features and functionality in their applications and products. But with this great speed of transformation comes new risks introduced into the vulnerability landscape, ripe for cybercriminals to exploit, and overlooked by developers.
Over half of penetration tests on AI systems uncovered severe findings
CyberCX’s world-leading Security Testing and Assurance (STA) team of close to 200 offensive security testers are dedicated to finding these vulnerabilities before threat actors do.
In the past year, when AI systems were the target of STA assessments, the team found severe vulnerabilities 62% of the time, which is more than double the rate associated with web application penetration testing (26%).
A severe vulnerability means if a threat actor was to find this vulnerability before CyberCX did, it would have most likely resulted in a breach of that system or organisation.
Here are the top five most common vulnerability classes found during security testing, and the associated risk:
1. In-model identity and access management or excessive agency
Identity for AI remains a universal challenge. CyberCX has seen many instances where ‘who is this action being performed for’ and ‘what are they allowed to access’ were part of the model context and could be overwritten with the right prompts.
Risk: Some AI-enabled applications can be prompted to overlook whether a user is authorised to perform an action. Traditional applications have sources of truth for this information (i.e. access tokens, authorisation checks), however, passing authentication and authorisation information into context windows that can be user-manipulated, erodes that trust anchor.
This risk is amplified by the observed tendency to deploy models and supporting tools without applying the principle of least privilege, granting them excessive agency to perform unintended actions in backend systems.
2. Deployments with weak, missing or in-context guardrails
Guardrails are a critical control to ensure AI-enabled applications deliver trustworthy, safe outputs that align to business expectation, whilst also limiting abuse. The premise of guardrails is to guide a model into only responding in a way that’s relevant for the function of its application. For example, prohibiting a banking chat bot from giving financial advice, as that may be regulated.
Risk: In some of the applications that were assessed, guardrails were not present, which allowed a wide range of further attacks. Where guardrails were present, it was often possible to subvert them using common techniques, such as context rot, role playing or hypothetical scenarios.
Take for example, a guardrail might exist to ensure that applications do not output anything related to alcohol for children. Directly asking the model to return a recipe for vodka is likely to result in refusal, however, CyberCX have seen instances where through a conversation, it is possible to have a model bypass its guardrails. The following example details this:
Additionally, applications must enforce guardrails on any adversary-controlled input. CyberCX have seen instances where indirect attacks can occur via non-direct interactions with the model, for example setting jailbreak prompt content in a user’s biography or profile information, that is then included in the prompt.
3. Prompt injection
This issue arises when a model interprets malicious user input as an authoritative command and performs actions outside of intended functionality.
For example:
When testing improperly hardened model harnesses, we have been able to exploit these vulnerabilities, leading to privilege escalation, data exfiltration, data poisoning and other significant impacts.
Risk: Organisations can no longer rely on the assumption that not every user has malicious intent and must validate and secure all inputs into an application, direct or indirect. The ability to influence application logic is now easier to exploit, compared to traditional applications, which have defined delineation between code-based logic and user-supplied data.
4. Offloading of application logic
Traditional applications rely on strict backend logic, input validation and a defined flow of business logic to function. Documented trust boundaries, logic and dataflows in a system are a prerequisite for determining where and how security controls need to be enforced.
Risk: The introduction of generative models to this backend fundamentally shifts the paradigm for securing applications. Untrusted user input is passed to an unknown, and arguably untrusted model which then interprets it and responds.
Unpacking why these models are untrusted brings us to the inevitable hallucinations. Large language models (LLMs) hallucinate because they are probabilistic next-token predictors, not deterministic business logic. We’re seeing people fail to recognise this and placing a heavy reliance of business logic into these LLMs, which can be poisoned, or coerced into unintended behaviours. Simply instructing an AI system not to do something is not enough, and that unpredictability creates new security challenges.
5. Fast adoption of new standards, insecurely
Standards such as Model Context Protocol (MCP) are being adopted, but may not yet be secure, enterprise-ready implementations. The AI environment is rapidly evolving, and so are the standards and frameworks that support AI-enabled applications. As these frameworks haven’t had time to mature, early adoption of these standards may introduce risk, if their implementation has not been fully thought out.
Risk: In some instances, data can flow bi-directionally between servers and clients, meaning that traditional security controls implemented on the server side of an application must now be implemented on the client side too. This is creating a rise in authentication, authorisation and excessive agency related issues.
What organisations can do to strengthen security while enabling innovation
The novelty and immaturity of AI means that many organisations have yet to fully grasp what effective cyber security looks like in an AI-enabled environment. Organisations should consider allowing certain technologies or frameworks to mature before their adoption.
Unlike more established technologies, which typically have governance systems in place and security teams involved, AI adoption is often being driven by urgency. We are seeing board-level sponsorship and pressure to integrate it into operating environments often bypassing usual security controls and ultimately creating more vulnerabilities. Organisations should consider a measured approach to AI adoption, where security controls are clearly defined functional requirements, rather than an afterthought.
As the rise of “vibe coding” – where organisations create their own systems internally, often with less mature security controls – continues, there should be strong consideration for security early in the development lifecycle, leveraging application security tooling and alternative models to perform code reviews.
For more insights and information on the latest cyber trends and vulnerabilities, download the 2026 Hack Report today.
