AI Agent Governance:
Inside the Glean AWARE Framework (with Cvent's CIO and CISO)

Michael Krigsman: AI agents are making decisions in the enterprise, but who's governing and managing them? Glean's Work AI Institute developed the AWARE framework to tackle agent security, governance, and compliance. 

Pradeep Mannakkara, CIO of Cvent, and Ben Mayrides, their CISO, explain.

Pradeep Mannakkara: Cvent is a leading SaaS hospitality and event software company. About 30-plus thousand customers, 5,500 employees, over 6,000 agents, and since about 2023, we're part of the Blackstone portfolio.

Security architectures weren't built for this 

Michael Krigsman: Ben, when we look at agent governance, agent security, this is a proliferating bundle of issues. What are some of the core challenges from your perspective as a CISO?

Ben Mayrides: Number one is education. The space is moving so rapidly, it's evolving so rapidly, and understanding not only what these tools can do for us, but also understanding what they can do to us, I think is incredibly important. So education, awareness, from the board level on down, I think is a big challenge. And again, it's just the rapid pace and the eagerness with which organizations, mine included, are adopting these technologies.

So that's, I think, challenge number one. But m- maybe to get a little bit more technical, I think that when it comes to some of the technical controls, the security architectures that we're all used to are really focused on deterministic systems. And as we know, AI is all about non-determinism. So the architectures are broken.

So when you think about agent identity, for example, agents are planning, they're reasoning, and they're delegating actions. And our existing IAM controls aren't necessarily architected and built for that right now. Same thing for observability, for example. You know, a lot of agent reasoning is opaque, and it's difficult to sort of reconstruct those elements as well, and we need that.

We need that for compliance reasons with things like the EU AI Act. But also in the case of a security incident, we need to be able to restructure and understand what actions an agent took, and so on. But the existing security controls that we have for observability aren't yet attuned, and they don't really integrate very well in an agentic architecture. So we need...

That's where we need purpose-built governance, we need education, and we need these technical controls to mature.

What agents do for us and to us 

Michael Krigsman: Pradeep, Ben made a really interesting comment. He said, "Agents are valuable for what they can do for us, but also to us." Maybe from a CIO perspective, dig into that. For us and to us.

Pradeep Mannakkara: Whatever we implement, we need to make sure there's adequate safeguards in place to manage risk, and I rely on Ben, our legal team, our privacy team to kinda help sort of shape that aspect of it, but at the same time, move fast. Now, your question around what it can do for us, I look at in terms of the value that it can bring us.

Whether it's taking action, whether it's finding information, whether it's suggesting, all those things have been highlighted, I think we've all seen where agents can do really well, and we actually have demonstrated ROI proof of that. The things that you can sort of take to us, don't know if I'd necessarily consider it negative, it just means we need to make sure we've got some mitigating controls in place.

And that can happen in a variety of ways. One of the challenges, go back to your last question, we have is there's a lot of vendors, partners, solutions that are out there. You multiply that times the number of employees we have, and now we have a lot of ideas, amazing, great ideas coming to us. And what we've got to figure out is how do we review these things?

How do we look at the value of each of these things? And as we try to do this in a meaningful manner, one of the things that we also look for is to make sure that there's financial safeguards as well in place. And then the 2 things is, you know, I rely on Ben and other teams to make sure we're protecting ourselves. That's how I look at the 2 part.

Applying the AWARE framework 

Michael Krigsman: You spoke about the various ways of looking at governance, looking at protections. The Work AI Institute's AWARE framework covers 5 governance challenges for agents: intent, context, guardrails, risk scoring and blocking, and ecosystem observability. How do these principles map onto your decision-making as you're deciding what to work on, what not to work on, what to hold back, and so forth?

Ben Mayrides: It's very relevant, of course, to any sort of agentic architecture that we have. So, you know, as we actually have been adopting the platform, it's been taking a look at, all right, what is an agent actually... what is the identity that it might be assuming or that might be delegated to it, and it's the who and what is taking that sort of action.

We need to actually identify that in advance, in addition to what is the scope that that agent is actually gonna run in? What are the data connections or the data sources that it's gonna run in?

And the context is incredibly important because, say, an agent that is maybe going to Salesforce and reading a customer record for the sake of summarizing that record for a salesperson versus an agent that is accessing that record to maybe update it, modify it, or maybe write, you know, summarize that and maybe write some of the sensitive data to another resource, that's a completely different scope, completely different risk profile.

So stepping through the AWARE framework, we're able to actually break that down and then make sure that we have the right kinds of controls in place.

Michael Krigsman: So it's essentially a comprehensive set of controls and sets of issues you need to look

Ben Mayrides: at. In my view, it's a technical structure, it's a technical control structure because, you know, the frameworks that are out there are predominantly organizational governance frameworks.

So when you think about the EU AI Act or the NIST Risk Management Framework, again, they're great and they have their purpose, but they don't go deep enough and they don't actually tie into sort of the technical controls and the technical considerations that you need to build into an agent tech security architecture.

Choosing platforms with built-in controls 

Pradeep Mannakkara: And I agree with Ben. So which is why sort of what we look for is try to figure out what's a good paved road of technologies that we can provide to our business.

I'm gonna separate our, call it our CTO organization, our product development organization, 'cause there we have a, I'm gonna use the word, a mature framework in place with actual appointed people and their roles, and they work within this framework that we've put in place. On the business side, we see a lot of desire to move fast. I see some heads nodding in the room.

And what we look for is there's a lot of demand that's coming to us to say, "Get us something quickly." So one of the things that... where we look to look for is technologies or platforms that have this sort of inherent within that framework or that platform or tool that allows us to then deploy something quickly.

'Cause we are here with Glean, one of the things that we realized with Glean as we kind of did that pilot initially and then rolled it out, was that actually bought us probably about nine months of time, candidly speaking, which satisfied a lot of our business users' needs, meaning they didn't want to rely on IT to do things, they wanted to kind of go and experiment and do things on their own.

And we as IT and security didn't have a facility to easily enable that within a corporate ecosystem, meaning can I get access to my email, my Slack messages, my Box files, Salesforce with that fine grain security controls and things like that.

So by us having the number of platforms that allow our users to achieve that, and then training, which you brought up earlier- Yeah, which then also security has sort of blessed the sort of the walls around it saying, "Hey, if you do it within this space, you don't have to come to us. We know the controls exist. Off you go," actually gave our users a ton of capability, to the point where we had a AI council with numerous requests that we would have to somewhat custom do. Those all sort of went off on the wayside. Now there's sort of the next evolution of that that's coming, which is I want workflow but...

Making governance a cultural shift 

Michael Krigsman: Ben, you mentioned the term training. Pradeep, you said surprisingly to me that culture is part of your role as- CIO. How do you make the framework come alive, which seems to me to be a cultural aspect? And it's more than just training. It has to be more deeply in- All right, so talk about that. How do you bring it alive?

Pradeep Mannakkara: We're very bullish on adopting responsibly using AI, you know, with all those other things that you talked about from a governance point of view. And as we're going through that, I'm like, "We're very bullish on this." However, we haven't baselined our employees. We've... maybe we have 15%, maybe I'm being generous, I'll say 20%, that for whatever reason have taken training, they've learnt it on their own, they're using it in their lifestyle.

But my belief is, you know, you might have a third, a third, a third. A third that are interested, a third that's like, "I don't believe in this," a third's like, "Let's see where this thing goes." But one of the decisions we made was we're gonna give mandatory training to everybody in the company, a foundations and a literacy.

And in there, security has a piece, legal has a piece, everyone has a piece around the education that kind of talks about some of the pitfalls with AI as well. So again, it's not a one and done, but it sets the sort of guidelines. Actually, our CEO was in the first session. Just can't tell you how important this was.

Michael Krigsman: So Ben, it sounds like this is fundamentally a different kind of training than we're rolling out a new tool and here's how, you know, here's how to use the tool. This sounds different.

Ben Mayrides: It is a big mindset shift, I think, as many would probably already understand and have experienced themselves. But I think in addition to what Pradeep was just saying in terms of that education and that cultural shift, you also have to have a structured risk process.

So as ideas, you know, this mindset shift happens and ideas, new ideas are coming up and new proposals are coming up from all over the business, we have to have a structured risk process to be able to take those ideas, evaluate them, yes, across a number of different dimensions, including security. And we have to have clear success criteria.

For evaluating and building into those projects, building into those deployments, and building into the operation of those AI systems.

Earning trust through deliberate risk decisions

Pradeep Mannakkara: Also, the respect that sort of the CISO organization earns, it has to be earned, that trust. And I think we had a recent example. Claude CoWork is out. Still in sort of the... it's not GA, but it's like, hey, it came out, people hear about it, they're like, "We want to test it." And-

Michael Krigsman: And Claude gives you messages when you open it up, "Hey, Claude is here. You should try this

Pradeep Mannakkara: now." One- 1,000%. And they also tell you, "Don't use this in a regulated environment," and so on. However-

Michael Krigsman: Yeah,

Pradeep Mannakkara: read the

Michael Krigsman: print the

Pradeep Mannakkara: fine print?

Michael Krigsman: Oh, and by the way, we're not responsible.

Ben Mayrides: Right.

Michael Krigsman: Yeah.

Pradeep Mannakkara: Now, at the same time, this is where the CISO, CIO, where I'm like, we should have a framework to evaluate that- Mm-hmm ... to understand capability, but not at the detriment to the company. Ben and team kind of looked at this and said, "Risk is too high." And not just risk is too high as a generic statement, but here are the reasons why, what

Ben Mayrides: those Well, and it's risk is too high for now. Because, you know, there were s- there, I think, a challenge that security folks run into is that sometimes maybe we're not looking at, the business, let's say, is not looking at all of the risk factors. Maybe, you know, traditionally or up to now, it's been very common for the business to look at model security.

Where it's about what data is that model trained with? What is the way to access that model? Is that model provider actually gonna retain my data, for example, and train, and use it to train their model? In an agentic architecture, it's completely different. It is about what that agent can do, what it can access, what decisions it's making, what actions it's delegating, what users it is impersonating.

It's all of those things. And so, whoa, let's kinda it's, so it's not no, it's let's take a look at this,

Pradeep Mannakkara: and-

Ben Mayrides: Let's

Pradeep Mannakkara: be deliberate about

Replacing gut reactions with shared criteria 

Ben Mayrides: it let's be deliberate about it. Let's be thoughtful. And again, you know, kinda coming back to the AWARE framework, what I like about it is that, you know, it sets that clear success criteria, for me and him in particular, but also other business leaders that are really keen to adopt these tools.

We can say, "Well, look, if we can just answer these, let's say, 5 questions related to the AWARE framework, if we can agree on and we can get to the same answers, it eliminates a lot of the... 'Cause this guy gives me a hard time all the time about, "Well, why do you not like it?" I'm just like, "It's a gut thing." And that's not an acceptable answer.

I mean, you know, would figure.

Pradeep Mannakkara: as

Ben Mayrides: you know-

Pradeep Mannakkara: therapist, what I tell

Ben Mayrides: him, it's

Pradeep Mannakkara: about the

Ben Mayrides: messaging. Right.

Pradeep Mannakkara: It's, it's how you message that same response

Ben Mayrides: can be interpreted.

Pradeep Mannakkara: Yeah,

Ben Mayrides: those answers get to a certain level of specificity, and we've agreed on the rules. He hates it when I say rules. But…

Pradeep Mannakkara: Guidelines ...

Ben Mayrides: guidelines.

Michael Krigsman: Frameworks.

Ben Mayrides: Requirements.

Michael Krigsman: Helpful hints

Ben Mayrides: Perhaps. But if we can agree on what the answers of those questions ought to be in advance, gets us to a level of specificity, removes the ambiguity of the, you know ... and that, from my perspective, gets me a little bit closer from and out of the realm of being a gate and maybe just being a part of a business conversation and a business decision.

Managing the speed-versus-safety tension 

Michael Krigsman: If I can paraphrase or simplify your roles, correct me if I'm wrong. Pradeep's job role as CIO is to move fast. Your job

Ben Mayrides: to- Like I said, is to make sure he doesn't trip while he's got the scissors.

Michael Krigsman: Okay. So we can agree. How does a framework such as AWARE help manage this really fundamental tension? I am sure you must have users coming to you, and they say, "We got to do this. You know, we got it now. We got to do it." How do you manage the tension?

Ben Mayrides: I think it's along the lines of what I was just trying to get at, which is that it gives us a I mentioned a little bit earlier having a structured risk process and clear success criteria.

It gives us a set of questions that if we answer those questions, then we can, you know, when it comes to being able to green-light a project, in particular, everybody knows what those criteria are in advance. They can build it into the design and to the deployment and the operation of those things, and we can get to yes a lot faster.

Filtering AI demand through ROI 

Michael Krigsman: Pradeep, when your users come to you and they say this- "We need this now," what do you tell them?

Pradeep Mannakkara: Similar to the sort of AWARE framework, we've also actually partnered with our PMO office and actually created an AI sort of flow, which thanks to our partnership with our finance organization, says, "Hey, you know, we If you want to go and demo something, by all means, go talk with the salesperson. Go see a demo. Understand what the capabilities are. Love it. Actually, and inform us.

I don't have to come up with the idea to implement something. We are actually, we'll learn from each other. Now, you want to take it a step further means that, ooh, there's value, and I love value. So that means if we're gonna help add to the top line or take away from the bottom line, however it is that's gonna drive value, I'm in.

Michael Krigsman: So we can do whatever we want, in other words.

Pradeep Mannakkara: With our partnership, 1,000%. So what, where that comes into is, great, now partnership with the CFO organization, you've got to have some sort of ROI if you're talking about some new investment. And if you believe there's value, we're gonna listen. And I've actually devoted resources specific to some of these AI initiatives so that we don't have those backlog with the rest of what's happening in the business. In order to move fast.

So that does not do with security. That's just reorganizing ourselves so that we can be much more nimble to get to these different gates, which is just to understand, hey, is there an ROI? Is there value? Is this thing real? 'Cause there's a ton of vendors and solutions that are out there. So that funnel within the business teams will naturally filter out before it even has to get to Ben or some of our teams.

Stage 2 of this is, I love to use the word test and learn. It doesn't mean full-blown production. It doesn't mean production data. It doesn't mean integrating it with the rest of our ecosystem, which has higher risk. And, you know, when you look at the checking the boxes on that framework. So we're like, "Hey, if it's a sandbox, let's create a workspace maybe that's not connected to something to test that out.

Let's make sure we have proper criteria." So not to kinda drag on, but what we've done is created different buckets based on the type of proof of concept or technology that you want- And once you get past that, that gives security time, legal time, privacy time to really vet these things based on those frameworks around risk that Ben and team have.

Shared language for hard tradeoffs 

Michael Krigsman: Ben, does the shared language that emerges from a framework like AWARE make it easier to work with your counterpart?

Ben Mayrides: For sure. So again, it sets up

Michael Krigsman: criteria- and to ease that tension?

Ben Mayrides: It... Well, I don't know. Okay.

Michael Krigsman: Let's

Ben Mayrides: not

Michael Krigsman: go

Pradeep Mannakkara: that

Ben Mayrides: far. far.

Michael Krigsman: Right. Yeah.

Ben Mayrides: When we're both under pressure, you know, to deliver and to help the company transform and adopt and sort of reimagine itself leveraging AI, that's a lot of pressure I think for anybody. But yeah, it again gives us that sh- I think I like how you phrased that in the question.

It gives us that shared language where, you know, a user comes to us and says, "Yes, we've got to adopt this tool now." And then I can say, "Well, look, yeah, so on device inference, that's awesome, maybe from the standpoint of it's going to maybe slightly reduce the risk of data exfiltration. But there... It opens up other cans of worms,"

and I can actually refer to the AWARE framework and say, "Well, you know, the problem is that I'm gonna lose potentially some observability, the ability to monitor, the ability to actually sort of reconstruct what might be taking place with an agent's reasoning, for example, if all that stuff is going on an end user's laptop.

We've got to figure out, also not only if we can monitor that and say if there is an action that it, that agent might be planning to take and we don't want it to take and we want to intervene before it actually executes anything. You know, we need that observability to be able to enable that kind of thing."

So again, it helps me to just have a sort of a logical way to step through and, you know, along with Pradeep and other technology partners in particular, be able to talk through these issues, and I like sort of the independence of it 'cause it's not just Ben thinks.

It's there are a bunch of other smart people out in the industry have given this some thought, and, you know, that I think as a security practitioner, if there is any in the room or, you know, online here, sometimes you have to sort of say, "Well, look, this is the way that other companies do it, and this is the way the industry does it." That does have, you know...

It is a little bit of a credibility.

Pradeep Mannakkara: I’m half joking, but if Ben's bot would give that answer- it would actually... People

Ben Mayrides: are,

Pradeep Mannakkara: "Oh, okay, the bot, AI bot told me it was too risky," versus, Ben.

Ben Mayrides: I tease, Well, I'm on camera, so,

Pradeep Mannakkara: you know, I got to- There is a little bit of, you know, I call it half joking, but people do look to the responses that AI will generate. And actually, Ben's team among other teams have actually created AI bots to kind of do some of that risk assessment, which think is one of the components of that framework as

Michael Krigsman: well. We've just spoken about how you overcome the fights.

Ben Mayrides: Mm.

Go/no-go decisions are never one and done 

Michael Krigsman: What are some of the real challenges when you're working together to make a go, no go decision about a model, a tool, what have you?

Ben Mayrides: When we're evaluating a go, no go decision and degree like something, is a tendency to look at only part of the problem, and you want to take as holistic an approach as you can. So I mentioned, some people, they only know model security. They don't know the agentic security and the implications of that.

The other thing about go, no go decisions is that I think especially given how fast the AI space is moving, how the volume at which we and other companies are going to be adopting and building agents and integrating them together into, you know, massive workflows and all that stuff, the technology is gonna evolve, the use cases are gonna evolve, the agents are gonna change.

There are gonna be new use cases for them. There are gonna be, you know, new delegation chains that are gonna pop up. So the go, no go decision is not a one and done, is my point.

Michael Krigsman: It's iterative.

Ben Mayrides: It's iterative, and it has to be streamlined. And you know, so he and others that are the builders and that are really focused on delivering, they need to understand that they need to come back to this risk process, and this is where he's like, "Oh, man." But

Michael Krigsman: it was good. I

Ben Mayrides: was gonna

Michael Krigsman: say- my

Ben Mayrides: job

Michael Krigsman: is

Ben Mayrides: to make sure that I can do that risk process at speed and at

Pradeep Mannakkara: scale.

Michael Krigsman: So you have-

Ben Mayrides: With clear criteria.

Michael Krigsman: So you heard what he said. I mean, you have to go back

Pradeep Mannakkara: and- Actually, we've got a better path, which is we're gonna basically have a catalog that he can

Ben Mayrides: There you go. Exactly.

Pradeep Mannakkara: Right. So this is how do you simplify-

Ben Mayrides: Yep

Pradeep Mannakkara: to Ben's point. Ben and legal and privacy have a need to know, hey, what is going on in terms of your organization, your ecosystem with regards to AI? So there is a documentation step that is an important piece if you ha- especially if you're regulated and things like that. That's going to become more of a requirement, especially is it customer-facing? What type of classification it is.

So a lot of those elements of that AWARE framework are actually going in. So one of the things I'm pushing or actually requiring is not at a software level.

We already have a software catalog if it's AI or not or whatnot, but a task level, activity level catalog of what is an agent doing, which the business or tech or whoever can essentially document, which then enables privacy, legal, security to query or have an agent go against it, and then based on risk, do what they need to. And if it changes Ben,

Putting AWARE into practice 

Michael Krigsman: What's your advice for CIOs, CSOs to put a framework like AWARE into practice?

Ben Mayrides: Consider the bigger frameworks, the more well, monolithic organizational frameworks that I mentioned, and synthesize a framework like AWARE with them. Inventory your agents. Instrument them for observability and identity. If you can only do, and focus on any part of the AWARE framework, it would be focus on those two things first, and then go from there.

Michael Krigsman: This is great. Pradeep, how do we put this into practice? What should we do?

Pradeep Mannakkara: People need to partner together, but you also have to simplify the messaging. The business folks want to know yes, no. Can I do it? Can I not? Why not? How quickly? So you've

Michael Krigsman: got to have- Yeah, we business people can't understand a whole lot more than that.

Pradeep Mannakkara: And to some extent, nor should they.

Ben Mayrides: Mm.

Pradeep Mannakkara: So that's the middle ground I think that's important for CIOs and CSOs, and as we work together, how do we help translate, simplify that message, give a framework that says, "Here's what you can do. Here's what we need to work together to enable"?

Michael Krigsman: How is this agent governance landscape going to evolve over the next year or two?

Ben Mayrides: So from a compliance perspective, I can see us seeing a control criterion in SOC 2 within, say, the next year and a half, two years. We need to be ready for that.

Pradeep Mannakkara: I would love to see vendors playing in this governance phase, so I don't have to work with Ben and team to configure that every single time we have different frameworks. I want an overlay, kinda like you do with security, that can work across different enterprise grade players. So enterprise-grade is gonna be important.

Michael Krigsman: Great. Do we have time for questions?

Scaling to 6,000 agents 

Audience Question: When you started, you said how many agents? 2,600?

Pradeep Mannakkara: We have, have over 6,000.

Audience Question: 6,000, okay. So when you scale them, starting from 10, 2,000, 100, what were your scaling challenges as you scaled to 6,000?

Pradeep Mannakkara: A lot of these. Now, we have different platforms where agents exist, which, you know, Ben's gonna make sure they're all good to run, but a lot of these agents, the sprawl, so to speak, happened because of the Glean platform, and it enabled them. Now, the Glean platform is one of the, all the different data sources and things that we have because of the fine grain security controls.

We trust it, and the users are trained to run. Now, one of the things, actually we talked a little bit about this, is when that started, we knew we were gonna have more agents. Now even though we have 6,000, how many are actively used? It's probably about 1,300, candidly speaking. But what we wanted to do was we need to change that culture.

We wanted to get people's behavior to understand the art of the possible. So we encouraged people to go ahead and create agents. Figure out how your life could be different. So they went on and started creating agents, started tweaking agents, learning from them. It was purposely done. Now, we knew that this was going to be a bit of a mess, but the purpose was different. The purpose was get people interacting.

Now, meanwhile, we also worked with Glean, said, "Hey, we need a way to moderate this stuff 'cause it's going to be a hot mess." And I call it within a 3, 4-month time, the capability came in. We can also see metrics and what's being used, what's not. Now the moderation is in play.

So part of this is not to say how do we stay one step ahead, but it's what's the goal at this point in time? It's a journey. Part of that journey was get people to create them and understand what's ca- and get them excited. Now, which we may be a long-winded answer here, but what we're getting now asked for is people want to create workflows. They're like, "Hey, great. I get the output.

Now I want it to take action." And they're very hungry to go to that next stage.

Michael Krigsman: So you have agents, and then they have kids, and they move out to the suburbs, and then before you know it, you have agentic sprawl.

Ben Mayrides: Yep. Great analogy. You know, the space is so dynamic right now. Again, you have to have a good development process, a structured and scalable, you know, tiered risk process as well because I think if you're applying governance to the same level of governance and control to every agent or every AI system out there, your governance capabilities are just, they're gonna crumble. They're not gonna scale.

Michael Krigsman: Okay, and with that, Ben Mayrides and Pradeep Mannakkara.

Pradeep Mannakkara: Thank you.

Ben Mayrides: Thanks, guys.

Michael Krigsman: Thank you, guys.

Pradeep Mannakkara: Thank you, Michael. Good job, Ben.

Ben Mayrides: Thank you, Michael.