Understanding the Security Risks of AI Applications

Dor Sarig

and

January 4, 2024

min read

Artificial intelligence is rapidly transforming software applications, but integrating AI models also introduces new risks. As highlighted in a recent analysis by NCC Group, AI-powered apps need an augmented threat model to fully understand and mitigate emerging attack vectors.

AI Models as Assets, Controls, and Threat Actors

Machine learning models occupy multiple roles within an application architecture:

Assets that contain valuable data and logic
Controls intended to make security decisions
Potential threat actors that could execute malicious actions

This unique positioning creates opportunities for novel attacks exploiting the AI components.

New Attack Vectors Targeting AI Systems

The analysis identifies numerous new attack vectors that threat actors could leverage:

Prompt injection - Attackers modify model behavior by injecting malicious instructions into prompts. For example, overriding safety clauses.
Oracle attacks - Blackbox models are queried to extract secrets one bit at a time, similar to padding oracle attacks on crypto.
Adversarial inputs - Slight perturbations to images, audio or textcause misclassifications. Used to evade spam filters or defeat facial recognition.
Format corruption - By manipulating model outputs, attackers can disrupt downstream data consumers that expect structured responses.
Water table attacks - Injecting malicious training data skews model behavior. Poisoning attacks have corrupted AI chatbots before.
Persistent world corruption - If models maintain state across users, attackers can manipulate that state to impact other users.
Glitch tokens - Models mishandle rare or adversarial tokens in unpredictable ways.
Many traditional vulnerabilities like CSRF also translate easily to AI contexts.

Architecting More Secure AI Applications

To mitigate risks, architects should design security controls around AI models as if they were untrusted components. Key recommendations:

Isolate models from confidential data and functionality
Validate and restrict model inputs/outputs strictly
Consider trust boundaries and segment architecture accordingly
Employ compensating controls alongside AI components

Even though it can be challenging, using a careful threat modeling approach helps organizations take advantage of AI capabilities while managing risk. As AI becomes more widespread, it's important to understand these unique threats and controls. Organizations that embrace threat modeling will be able to create safer AI-powered systems.
Pillar Security provide expert threat modeling services for AI solutions used in mission-critical business applications. Contact us to learn more.

‍

FAQs

Why do ML models require a different threat model than traditional software components?

ML models occupy three distinct roles simultaneously: they are assets containing valuable data and logic, controls that make security decisions, and potential threat actors capable of executing malicious actions. This unique positioning creates opportunities for novel attacks that traditional threat models were not designed to address.

What is a water table attack and how does it affect AI systems?

A water table attack involves injecting malicious data into a model's training set, which skews the model's behavior over time. Because the corruption happens at the training level, the resulting bias or vulnerability can be difficult to detect once the model is deployed.

How do oracle attacks work against AI models?

Oracle attacks target blackbox models by querying them repeatedly to extract secrets one bit at a time. Even without direct access to the model's internals, an attacker can systematically probe its responses to piece together sensitive information.

What is the risk posed by glitch tokens in AI applications?

Glitch tokens are rare or adversarial tokens that cause models to behave in unpredictable ways when encountered. Because models mishandle these tokens, attackers can potentially exploit them to produce unintended or harmful outputs.

What is the core architectural principle recommended for building more secure AI applications?

Architects should treat AI models as untrusted components by isolating them from confidential data and functionality, strictly validating and restricting their inputs and outputs, and segmenting the architecture according to trust boundaries. Compensating controls should also be employed alongside AI components to manage residual risk.