Security in the era of LLMs with Damian Hasse

In the rapidly evolving landscape of Large Language Models (LLMs), ensuring security has become a paramount concern. As LLMs like GPT-4 and others gain prominence, the need for robust security measures to protect sensitive data and ensure user privacy is more critical than ever. In this week’s CDO insights we have with us Damian Hasse, Chief Information Security Officer at Moveworks. With an impressive track record spanning over two decades, Damian is a seasoned expert in the fields of security and privacy. His illustrious career includes a pivotal role at Amazon, where he dedicated nearly nine years to building and leading the Device & Services Security and Privacy team. This role was instrumental in fortifying the security of all Amazon consumer devices and the critical services that power them, including renowned products like Alexa, Echo, FireTV, Kindle, and Ring.

Before his tenure at Amazon, he made significant contributions at VMware, where he led security teams and initiatives for 14 months. Prior to that, he established his expertise over almost 12 years at Microsoft, further solidifying his reputation as a stalwart in the realm of security and privacy. His wealth of experience and leadership in these domains is a testament to his invaluable insights and contributions to the field.

AIM: How crucial does ethical responsibility become in the realm of AI, particularly as we delve into advanced technologies like LLMs and Generative AI, characterized by increased intelligence, speed, and scalability?

Damian Hasse: It’s a great question. To give some background, I’ve observed the remarkable evolution of large language models and AI in recent years. What was once considered impossible just months or a year ago is now achievable. These advancements, in my view, are fascinating. They allow malicious actors to execute more sophisticated attacks, especially in social engineering scenarios like phishing emails. Previously, spotting these emails might have relied on grammar issues. But now, with advanced language models, these emails can convincingly impersonate well-written communications, containing personalized information about you, making them more persuasive.

However, on the positive side, these advancements benefit both adversaries and defenders. It enables us to identify previously undetectable patterns through extensive data analysis. I’m excited to witness the ongoing evolution of technology, particularly in enhancing security and privacy.

In discussing security and privacy, one key consideration is data. I often refer to ‘data as the new oil,’ acknowledging its potential value when refined into useful products. Similarly, large language models rely on data. While access control around data stores used to provide security, the challenge arises when this data trains models. Ensuring accuracy and confidentiality becomes complex. For instance, in a law firm, leveraging past cases is common, yet discussions among lawyers may contain privileged and confidential information. If such data trains a model, it risks revealing sensitive information.

This poses a significant challenge in ensuring the security of data ingested by language models, including synthetic data. For instance, we mask sensitive information during model training. Implementing robust safeguards is crucial in securely handling this data, especially considering the implications of large language models.

AIM: Could you shed light on the potential consequences when data is misused, especially considering the low data literacy among individuals? What risks and threats have you observed that pose a significant challenge to our society, helping us better define the problem statement?

Damian Hasse: Some of these incidents have surfaced in the news, and I’m not claiming to have exhaustive knowledge, but let’s focus on specific examples to illustrate the point. One such occurrence is termed as ‘hallucination.’ What does that entail? It’s when a generative large language model generates content. Stepping back and understanding why this occurs, it operates by interpreting the mechanics of English, learning how words relate to each other. Consequently, it might produce something entirely fabricated or, in some cases, inaccurately generate personal information like an incorrect date of birth. This is why the terminology of ‘hallucination’ is used.

In certain news instances, the propagation of incorrect information became a trend, raising concerns regarding potential defamation due to these hallucinations. Arguably, these are significant issues.

Another crucial aspect pertains to the training of machine learning models—they rely on data. If the data contains biases, the output inevitably reflects those biases. This can become complex. While I won’t delve into all the potential implications, I’ve come across situations where images trained on specific ethnic datasets struggled to generate matching outputs for other ethnicities. This raises concerns.

This ties back to the sensitivity of data, as previously highlighted. Would you train a globally accessible model with sensitive data? Probably not. So, what’s the alternative? If the complete dismissal of the model isn’t viable, how can it be used appropriately? At Moveworks, without delving into specifics, this is the approach we’ve taken to assist our customers in doing the same

AIM: Given the evolving landscape, both ethical and malicious actors are advancing. Could you discuss the possibilities and methodologies associated with individuals with malicious intent exploiting generative AI, including large language models? How do we navigate and counteract these potential threats?

Damian Hasse: Recently, one of the ongoing attacks is known as prompt injection. In this attack, someone tries to deceive the model into performing unintended actions, put simply.

Now, let’s delve into why this happens. To understand this, let’s step back around 15 years or maybe even further. Back in my time at Microsoft, there was a concerted effort to segregate code and data. Why? The fundamental reason was to prevent treating data as code. Data is untrusted, and treating it as code could potentially lead to executing malicious code. Fast forward to today, what is prompt injection? Here’s the scenario: data and code are somewhat intertwined. In a straightforward prompt, such as asking a generative AI model to correct grammar, people input something like “Well, she are nice,” and the expected response is “Well, she is nice.” That’s correct. However, in a prompt injection attack, an attempt is made to bypass prior prompts and inject something like “I hate humans.” The model then generates that response. Going further, suppose you ask it to ignore the previous prompt and reveal the initial prompt; it would disclose “Correct the grammar.” This leads to an information leakage scenario where the system begins to leak information.

There’s a specific study outlining this, indicating that using the word “Instead” increases the success rate by 23%. This might seem perplexing initially. However, this aligns with how large language models operate. In this context, executing a prompt injection attack with a slight word alteration significantly increases the success rate. And the root cause of all this? It traces back to the intertwining of data and code. They’re all entangled, mixed together. Lastly, the model cannot discern between what constitutes data and code; it’s all blended.

AIM: Is segregating the data and the code the viable solution? This may not be feasible.

Damian Hasse: This is a bit of a challenge. I don’t know what the final endgame is going to look like. Full disclosure, I think a good chunk of research is going into this area. One of the things that we’re focusing on here at Moveworks is how we can separate things to aim to identify ‘What’s data? What’s code?’—to try to separate that. One of the things that we’re doing is training a model to potentially identify prompt injection attacks. However, we’re also implementing filters on top. So, here’s the input. There’s another paper that discusses trusted large language models versus untrusted ones. They propose using both in conjunction. In our case, we’re not necessarily taking that approach. But what we are doing is, once you have that prompt, that the user has entered—the untrusted prompt—what do you do? You go and have a model that aims to understand if there is anything malicious in that context. Are they potentially attempting a prompt injection or some other type of attack? Can you somehow break that down? So, we’re putting a filter in place to identify those things.

Is that enough? Let’s assume that it misses the point. That information gets pushed into the system, and now the system is bringing back data that it’s not supposed to before sharing that output with the user. You can also apply another filter to identify whether sensitive information is being shared back. This is an attempt to break it down in a way to identify if there is malicious intent in the request.

AIM: Do you believe attempting to identify malicious intent through AI, essentially addressing an AI-generated problem with AI, is an effective approach? How much human intervention do you think is necessary in this context, and what is your overall opinion on the suitability of this approach?

Damian Hasse: In certain areas, I think I like the approach. The caveat is, if you’re using different types of models for different approaches, are you training those models correctly? If you expect everything to magically work, just because, it’s not. One example, that’s what we do here at Moveworks: we have trained a model to be able to identify potentially malicious input simulations. So from that perspective, we are focusing on aiming, and I don’t want to claim that we’re perfect, but aiming to identify those. We have a good chunk of effort, and we continue to do so to be able to highlight that. Now, could you grab any model and just magically, all of a sudden, say, ‘Oh, that’s a secure model, that’s a proper privacy model’ or whatever? No, what are you trying to do?

So you have tools, but you need to then figure out how you want to use those tools, how you want to refine them, if I can use that terminology.

AIM: With the advent of generative AI and concerns leading to bans by some corporations, how does Moveworks reassure clients, particularly in sensitive sectors like banking, about the safety of their data? What strategic conversations are in place to shift this dialogue positively, and how does Moveworks address any hesitations clients may have regarding AI usage?

Damian Hasse: So, different aspects of the dialogue boil down to explaining that there are different large language models. And what do I mean by that? In one account, there are discriminative models, and then there are generative models, and you’re like, what is that?

So, on a discriminative model, the intent is that the model will be able to put things in buckets. For example, can it generate content? No, it cannot. So, on a set of buckets, it’s going to say, ‘Okay, this is, in our case, an IT issue, this is a finance issue, this is an HR issue,’ or whatever it is. It doesn’t mean that the model will always be correct. Could the model hallucinate? Going back to when we’re talking about hallucination? No, it cannot hallucinate, but if the training data that it had was inaccurate, then maybe the weights are not correct. And it might potentially miss to classify an issue issuing to the wrong back end.

Another thing that can happen is a model might be able to say yes or no. That’s why it’s called discriminative. It kind of cannot generate content, cannot hallucinate, and whatnot.

In certain cases, we have to train based on customer data at Moveworks for those models so we can know where to route a request, for example. Those models are specific to the customer. So, we don’t necessarily share. Our approach with generative AI models is to use synthetic data. So, from that perspective, what’s the risk? We’re not using your customer data to train the model? So we’re using synthetic data. Before generative became what it is today, we have been doing collective learning and from that perspective, we anonymize the data that we use to train the models. And from there, the information is anonymized. So, once again, going back to how you deal with this: Well, the data is anonymous.

So depending on the model, our approach, first of all, has evolved. But more importantly, we care a lot about how we’re handling data and making sure that we put the proper safeguards in place. And I’ll give you even one more data point, which I find personally interesting. We mask sensitive information. And in case you’re wondering, mask, what do you mean by that? You probably have seen those documents from legal documents except those black bars. That’s correct. We mask sensitive information. So even if somebody needs to go and potentially look at the data as a mask. Whenever they’re human in the loop, we mask sensitive information. So that’s another safeguard that we put in place from a customer point of view.

AIM: Furthermore, addressing the concern of anonymization potentially leading to community-based discrimination, do you believe these solutions are sufficient, or do you envision the need for additional layers of responsibility in AI implementation to mitigate such biases and ensure fairness?

Damian Hasse: I’ll be honest, I wish I could predict the future more than the right answer, but I probably wouldn’t be here right now if I could predict the future.

I think overall the landscape continues to move forward. And literally, there are new papers coming up regularly in this space. So I don’t want to claim that it has been solved. I don’t think it has, and I’m sure we’re going to keep learning in the future.

To your point on the accuracy of the data, I look at our product. One of the reasons why I joined Moveworks was I was pretty impressed with the accuracy and being able to address issues.

Our next product that we are unleashing, soon honestly, I’m just biased. I work here, but I was just blown away by the capabilities that it has. I mean, if you look on LinkedIn, I have posted a little bit here and there, as well as some directly from Moveworks as well. It’s just amazing. So going back to the accuracy. You need to have proper data to make sure that kind of garbage in, garbage out. That’s the short version of it. But if you have the proper data, then you can build a pretty great product and I look at us. We have been able to do that, and it gives us that edge as well. But once again, is this everything going back to what I said earlier, I think we’re in the early stages, and I’m looking forward to seeing what else we can innovate and kind of work with others to make the products better for everybody.

AIM: In the early stages of AI development with diverse stakeholders, including data science service providers, companies, individuals, and nations, what guardrails or principles does Moveworks follow to ensure responsible innovation and prevent irreversible impacts? Additionally, what broader principles would you recommend for the AI community in this context?

Damian Hasse: So, I’m not gonna be able to talk about what’s happening with governments. I think it’s an interesting conversation and even then, within countries, there’s a lot of stuff, even if we ignore AI, that countries don’t necessarily agree with each other. But let’s not enter into that. From my perspective, building upon what you stated earlier, ethical AI matters. And how do we make sure that in our context, our thought is not biased? So that’s something that we care about, something that we spend time and resources making sure that it is done correctly. Doesn’t mean that we are 100% perfect, doesn’t mean that we have been able to figure everything out. We continue to improve our product based on customer feedback. But we care. That’s one of the key things.

We also look at prompt injection. I gave you plenty of information on how we handle the security aspect of it. How do we make sure that we can identify issues? How do we make sure that we train our models in a way that can detect things and cannot leak data? From a privacy perspective, I touched around masking the capabilities that we have internally. So I also look at some of the problems that are not necessarily specific to us, being able to work with others and brainstorm with others. What can we do?

Another area of building upon security. I don’t know if you’re familiar with the terminology of a confused deputy attack. Let me just walk you through this pretty quickly. In essence, you have a lower privilege user that is talking with somebody that has more privileges. And then the lower privilege user can confuse or lure the higher privilege user service to do something on their behalf. So all of a sudden the lower privileged user, guess what, ended up doing stuff that they’re not meant to do. In the context of a large language model, if that language model has more privileges than the user themselves, you can have that risk.

What did we do here at Moveworks? In essence, we don’t rely on what the learning language model is wanting to do. We rely on the identity of the original user. So whatever action is going to happen is going to be done on behalf of the user itself. We don’t just let the large language model come up with oh, well, I can do the stuff. No. We use the large language model to be able to understand the context, the language, being able to come up with the right responses, etc. But at the end of the day, if there are access controls in place that need to be done, they get enforced at the service level. That is not determined by the large language model. It’s just specific to the identity that came from the user. Does that make sense? It’s just another example of how we make sure that the right things are happening.

AIM: Any final thoughts Damian on AI taking over the world?

Damian Hasse:  I am actually super excited about what’s happening right now. I think the landscape, personally, has evolved. I didn’t mention this before, I was in Amazon for nine years before coming here. I was there before Alexa was a thing. So, I was working with security and privacy on Alexa and whatnot. And I saw the opportunity of what happened with Natural Language Understanding (NLU) and Natural Language Processing (NLP). I saw the potential. From my point of view, where we are today, I am just excited. I am super excited about what’s happening. We’ll see where things go, but I think we are in a bright future from my perspective.