Recent events in the news have us pondering a fundamental trust issue with AI models, whether that’s trusting the data with which we train them, or the results they infer. 

I might have missed something, but I’ve never seen a recipe for pizza that involves glue! Then again, I suppose an AI model wouldn’t know what glue tastes like.

Large Language Model (LLM) poisoning attacks that have made the news within the last year have proven that anonymous sources and automated training loops can wreak havoc on the trustworthiness of results, or the security controls implemented by an AI service.

Take ChatGPT, for example, one of the most popular AI tools as of late, it is trained primarily using public sources on the internet. It is also known to produce factually incorrect information. Some of this may be attributed to the confidence and accuracy of an AI model. Still, a model can also be trained on factually incorrect data, just as a grey-zone style data poisoning attack can adversely affect the dataset used to train a model.

Fundamentally, I see this as another zero trust problem. Zero trust in its abstract definition means that I should be verifying and scrutinising every interaction against a security policy, such that I can place trust in the system, data, or application I’m trying to access. See my previous blog article – Zero Trust is not just an architecture | Nexor – for an example scenario on using the principles of zero trust to withdraw cash from an ATM.

What do I need to know to have confidence that the dataset I’m training an AI model with is reliable? Similarly, what do I need to know to trust the response an AI model has inferred?

This is an illustration from NIST 800-207 – the Zero Trust Architecture standard. This illustration demonstrates how an implicit trust zone can be established from a starting point of zero trust, using a Policy Decision Point (PDP) to enforce a policy decision at the application - known as a Policy Enforcement Point (PEP)- driven by a relevant security policy.

The key part of the PDP/PEP architecture is that scrutiny of the interaction or artefact allows me to begin to trust the resource.

Theoretically, this approach could be applied to both training datasets and the inference result. Let’s take the training set for example. If I am producing a training dataset for an LLM such that the data needs to be trustworthy and reliable, I would want to ensure that I can guarantee the origin with a reasonable degree of confidence, of the data that I am placing within a dataset. Perhaps I might scrutinise the authors of the data to ensure they are likely to produce reliable data, and even so, I might require it to be cryptographically traceable to the author.

Similarly, if I am going to use a result that has been inferred by an AI model, I might want to place a level of trust depending on several risk factors. If I am concerned about the data that has been used to train a model, then I would want to understand the provenance of the data. Would I want something trained from unregulated, publicly accessible data to provide me advice for a topic requiring high assurance of being accurate? Similarly, I might want to know if an AI model is running software components that make it susceptible to being vulnerable to poisoning attacks, to determine if my answer could be compromised. I could go on with examples, but the principle remains, that there are considerations that should be made with results inferred from AI models, before they can be used in meaningful ways.

Provenance data refers to a documented trail that accounts for the origin of a piece of data. In software development, this might be accounting for how the software was developed, what toolchain was used to build it, and which processes it went through to be built. Several specifications and frameworks are actively trying to mature this, such as SLSA and In-Toto. In the AI world, this could be where the information to train a model was sourced, how it was produced, and perhaps how it is verified for authenticity. Provenance can give an indication as to whether the process used to achieve a result contains risks that can or cannot be managed. For example, I can’t trust an AI model to help me write a blog on history accurately if the dataset it has been trained on is an alternative history novel.

Based on some research I conducted with a public authority and industry partners involving software supply chain security, there is a similarity between the risks and potential risk management strategies with AI models. A generic case might be that I can’t understand the software bill of materials of some software, therefore, I cannot completely understand the risks that software presents in a safety or security-critical context.

Software supply chain risks are similar to some AI training dataset risks. How do I ensure a node package manager (npm) library hasn’t got a zero-day backdoor hidden inside? How do I ensure a picture from the internet isn’t poisoning my AI model to realise a disinformation campaign?

Is my supplier producing code that is following the required code guidelines for my security-critical application? Is the Wikipedia article I am training my model off accurate at the time I’ve trained my model (or has someone adversely edited it)?

The SLSA specification which aims to allow automated decisions about the integrity of software, defines the purpose of provenance as allowing consumers of provenance data to confirm that artefacts are built according to expectations and that a faithful reproduction of the software and its toolchain can be performed. If implemented at the beginning of a software project and used throughout the supply chain, it allows a chain of provenance information including software attestation to be created. That attestation can then be continuously and automatically tested in a DevSecOps environment, against a relevant security policy.

In real-world terms, the provenance of components, tooling, inference, and data helps the executive decision maker holistically understand the risks that are presented to my system or capability. When I understand the risks, I can choose to employ the appropriate risk management strategy.

If my AI model is trained on the contents of Wikipedia and my results reflect this, then I can manage the risk of data inaccuracy to the degree that I am comfortable with. Similarly, if my software has records that prove it has been developed to my requested guidelines, I can manage the risk of running that software in my safety-critical or security-critical environment. All of this comes with the caveat that I can trust the provenance data itself, but this is not an impossible task.

While there is no magic silver bullet to preventing a poisoning attack, provenance allows us to make risk based and quantifiable decisions about data – whether that’s software, datasets, AI, or the contents of a USB flash drive (or floppy disk) that you got from a random person at a conference or on the London Underground – and how we deal with it based on our risk appetite and tolerance. Supposing there is a reasonable methodology to being able to trust the provenance data to begin with, it would be a reasonable assumption that one could either control what data an AI model is trained with by deciding on whether to accept the inputs based on the provenance of that data, or whether a user should trust what an AI model is inferring.

In my view, the disillusionment people have experienced with AI is a direct result of placing blind faith in this magic technology, without understanding the challenges or risks associated with adopting it. While experiments are useful to develop technology quickly, the real-world usage must consider these risks up front to make sure they are managed from the get-go, to make sure that the realisation of these risks doesn’t cause damage – just like software. Understanding the risks up front empowers decision makers with the ability to leverage leading-edge technology in the systems they are responsible for, while delivering the assurance that said system is secure and dependable. This approach is the essence of Secure by Design – understand your risks up front and design your system to manage them appropriately. Empowering systems and people to make risk-based decisions based on data is a significant step toward reducing the risk of being compromised – whether that’s deliberately or accidentally. The NCSC have highlighted in their annual review that nation-state threat actors and cyber criminals alike are working furiously to increase the volume and impact of AI-based attacks. It’s crucial that we start with understanding how we need to manage these risks before we completely rely on these tools to deliver our future.