Snap:
Agentic AI Software Development at Scale

Saral Jain: At Snap, more than 90% of all code is written by AI agents right now. AI is not about a bunch of disconnected agents. It is a managed production system.

Michael Krigsman: At almost 1 billion users per month, Snap runs on AI agents. Saral Jain is head of engineering and CIO at Snap, and his team built those agents.

Agents write the code, engineers own it

Saral Jain: Our code reviewer, CodePal, reviews 90% of all of this code within the first 5 to 10 minutes, and it has found tens of thousands of bugs in new code that is being written. Casper, which is our remote coding agent, is generating thousands of PRs every month. At our scale, this is what agentic software development looks like.

Somebody comes up with an idea, the idea gets picked up by Casper, it spins up a remote sandbox, it does the safest, smallest possible change, generates a PR, CodePal reviews that code as an AI agent. They do their back and forth, and then a PR gets generated for a human who's the ultimate accountable owner to take a final look, and the PR gets shipped. So it's really about all of these systems working well together for a production agent.

Michael Krigsman: Give us some insight, a sense of what that scale actually is, and then I have a bunch of follow-up questions based on what you just were talking about.

Saral Jain: Snapchat reaches almost a billion people every month. So it's one of the largest global platforms in the world. So every change we make as engineers impacts and goes out to almost a billion people. That is the scale we are talking about. And behind the scenes, we have thousands of microservices running across thousands of repos that enable the backend for Snapchat.

Michael Krigsman: And when you talk about agents being managed, how do you manage agents? Can you give us a sense of the architecture of agents managing, agents working, and then agents managing other agents, and then the teams of agents? And, do you think of agents as employees, or how do you just think about all of this?

Saral Jain: The way I think about it is a golden path of agents. What we don't want to do is every employee do demos and prototypes and create a bunch of disconnected agents that do not solve the problem in a safe, secure, privacy-safe way. So what we have is we want to build this golden path of what engineering looks like at Snap. What does an engineer actually do? They generate code, they review code, they investigate issues, bugs, crashes, outages, look at A/B test results, and so on.

So we have identified what are the core workflows for engineers and then created these managed agents, the blessed agents that do the production way of doing what these workflows should be doing. That is our way of saying the safe way should be the easy way so that people actually use these managed blessed agents, essentially.

Michael Krigsman: We have a really interesting question on LinkedIn from Peter Chapman, who asks whether engineers still understand the code that is being created by these agents.

Saral Jain: The sheer amount of code being generated by agents now is so large that it is a very fair question to have. But ultimately, for us, engineers and humans are the ultimate accountable owners for any production system. Which means that they should read the code, they do absolutely understand the code, and they very much own the systems behind the code. And that's just how we think about our engineering culture is ultimate accountability is with the engineers and they are expected to understand all of the different ways the code changes essentially.

Michael Krigsman: This episode is brought to you by Gartner IT Symposium Expo. Ready to scale agentic AI from pilot to production? Join top CIOs and IT execs this October 19th to 22nd in Orlando, Florida. Over 300 Gartner analyst-led sessions will cover top priorities shaping IT, from AI value, governance, and cybersecurity to cost optimization, IT operating models, and beyond. Get practical insights and connect with peers tackling the same challenges you are.

Secure your spot today at gartner.com/us/symposium. Give us a sense of the number of agents, the kind of scale, and the number of employees that Snap has so we can try to merge these two sides.

Saral Jain: We actually have an agent inventory, and so the number of agents is thousands, but the production agents, what we call the golden path of agents, we have created these 14 agents that cover almost 90% of what an engineer's day-to-day workflow looks like. So those are what we call the golden path. But apart from that, we have thousands of agents running. Overall, Snap has roughly 5,000 employees. And so at this point, the number of agents will outgrow the number of employees pretty soon.

Michael Krigsman: So, given the number of agents and the amount of code that's being generated, how do you ensure that the engineers understand what is going on? Because I think we all see the temptation to accept what an agent provides to us and say, oh yeah, agent, that looks good, without fully understanding the nuances. How do you manage that?

Saral Jain: Yeah, it's definitely challenging because of the sheer amount of code being generated, but at the same time, it's ingrained in us that whatever changes we make in engineering go out to a billion people. There are so many edge cases, so many performance characteristics, so many device types that Snapchat runs on that we have a very, very good set of guardrails for any production change that goes out. These guardrails include human verifications, but also AI verifications. So as an example, one of our most popular AI agents is called CodePal.

This is an AI reviewer. It reviews all of the code that gets generated essentially and finds bugs in this code. Investing in evaluation in these eval layers in the code reviewer agents, as well as being able to quickly roll back when something bad goes on, gives us a lot of confidence to trust these agents essentially, and that is how we kind of think about agentic development. But I will reiterate that ultimately humans are accountable for all of the production systems and all of the code goes in.

So while CodePal generates the first layer of code reviews, while Casper generates a lot of the PRs, the humans will absolutely take a final look at these PRs being generated before anything gets merged into production. And that's just part of the DNA, the culture of the craftsmanship of engineering we have at Snapchat.

Agents as teammates with defined personas

Michael Krigsman: Folks, you can ask your questions. If you're watching on LinkedIn, pop your questions into the LinkedIn chat. If you're watching on Twitter/X, use the hashtag CXOTalk and be sure to me, @MKrigsman, on Twitter so that I see your question, but take advantage of this opportunity to ask Saral Jain from Snap pretty much whatever you want. It's a special opportunity, so use it. So, Saral, can you describe the kind of team-oriented architecture of how you organize these agents?

Saral Jain: Let me walk you through what a day-to-day of an engineer looks like. At this point, I've often seen that engineers start the day with a standup with their agents. They try to review what happened last night, what did the agent do, what are they firing off the next set of instructions to the agents, and then checking in on those instructions through the day during meetings, at lunch, on their phone, on Slack. So that is what orchestrating a bunch of agents looks like.

The other big shift we are seeing is how teams communicate with each other and how teammates communicate with each other. Casper is an agent that we have built. It's a remote coding agent. What teammates do is they're having a conversation on Slack. They're talking about a new design, or they're talking about a feature or a bug that needs to be fixed. Casper is listening in on the Slack conversation or on the conversation that is happening in Jira or anywhere else in our internal systems.

At one point, when you're done with the back and forth with your teammate, you can tell Casper, Why don't you go ahead and build this feature or fix this bug? Casper has the context of this conversation, but it also has the context of all of our codebase, all of the information at Snap. It spins up a remote sandbox, it generates the smallest, safest PR, and creates that PR. CodePal then reviews that PR, they do the back and forth, and then finally the PR is sent to a human for approval.

So that is what collaboration looks like right now. There is no more Okay, let's create a meeting in three days and we'll talk about it. The conversation is happening live and the PR is generated in that conversation, which I think is very fascinating.

Michael Krigsman: A lot of times, people talk about agents as being employees. Do you think of agents in that way?

Saral Jain: I wouldn't characterize it as that, but our agents certainly have an identity just like employees do. So we have similar rules of audit trails of what an agent does, or we have rules around permissions of what an agent, so think of agents as having a persona. What can this agent do? What data can this agent access? What is the network boundary? What is the set of other systems that this agent can read access to or write access to? What is the audit log? So we have built all of that.

So in a sense, yes, our agents have personas, but certainly our agents are not necessarily the same as our employees.

Michael Krigsman: What exactly do you mean that agents have personalities? You know, I'm asking all of these questions because you're so experienced, you're so far down that maturity and learning curve with agents that it's really helpful for us to gain insight into what you're doing. So, what do you mean that agents have personalities?

Saral Jain: Agents have personas. Personas define what an agent can do and cannot do, essentially. And so what we don't want is agents, thousands of agents running amok and creating chaos. And so we have defined a lot of guardrails in terms of what access, what system access agents have, and what changes can agents make. That is the personas that we have defined for agents. And different agents have different risk tiers.

For an internal system that is accessed only by certain employees that does not have sensitive data, agents actually can do the majority of the work. For an external system that impacts our Snapchat community of a billion people, the guardrails of the agents are much stricter. And so that is what the agent personas mean.

Michael Krigsman: This episode is brought to you by Gartner IT Symposium Expo. Ready to scale agentic AI from pilot to production? Join top CIOs and IT execs this October 19th to 22nd in Orlando, Florida. Over 300 Gartner analyst-led sessions will cover top priorities shaping IT, from AI value, governance, and cybersecurity to cost optimization, IT operating models, and beyond. Get practical insights and connect with peers tackling the same challenges you are. Secure your spot today at gartner.com/us/symposium. Let's go to some questions. We have some really good questions. Let's flip over to Twitter.

This is from Chris Petersen, who says, a billion users is an interesting statement today. Is that humans, bots, agents, and agents, or just humans? And, what is Snap's plan to support customer and member agents leveraging the platform from the outside?

Systems for agents, features for humans

Saral Jain: When I say a billion users, I mean Snapchat users. So people who actually use Snapchat every month to talk to their close friends and family, that is the core of what our app does. However, the interesting part of the question is also our users can also be agents, but for internal purposes. So we have often seen that internal systems like code search that was originally built for humans, now our agents have 60 times more traffic on those code search systems internally than humans.

And so when I think of building systems now, you have to think about building systems for the humans, but also for agents essentially. And so that is a very interesting insight into the question, and we are certainly seeing that happen internally as well, where our internal production systems are now being used by agents as much, if not more, than humans. In terms of supporting agentic capabilities for our Snapchat app, that is also something we have a lot of fun ideas on. But AI for AI's sake is not interesting.

What we want to do is build meaningful features on Snapchat that promote our core vision, which is how can you make a space that makes it fun to talk to your close friends and family, express yourself, and be in the moment. And so when we think of building AI features on Snapchat, that is the set of principles we always think about, not whether these are AI features or not AI features.

Michael Krigsman: It sounds like you're extremely disciplined in terms of having that very clear guiding principle of user benefit and then relating the technology to support that in your development process.

Saral Jain: 100%. Technology is here to help enable our mission, but technology is not the mission. Ultimately our mission is connecting people and providing a fun way for people to communicate, essentially. And when technology becomes a way to emphasize that mission, that is the most interesting space. Want to be.

Michael Krigsman: So, how do agents then support that mission of making Snapchat interesting and fun and useful?

Saral Jain: We are an app. Our ability to build and ship features to our customers defines how we can delight our customers, our Snapchat audience essentially. And agentic software development has meaningfully changed the way we can ship quality software faster. So that, I think, is the biggest change. Our per-engineer commits are up by 75% year over year while reducing the number of severe outages by 57%.

That is such a unique statistic that while we are building more features for our community, we are doing it in a safe, reliable, performant way for our community. And that, as an engineering leader, is music to my ears. Apart from that, we are also using AI to build interesting features for our community. We have built My AI, which is our AI assistant that every Snapchat audience member has access to.

And people use My AI for so many fun use cases that it's really, really good to see that AI is actually now being part of the daily life of so many people.

How Snap governs thousands of agents

Michael Krigsman: Let's go to some additional questions. You guys, ask questions! When else are you going to have this opportunity? So, take advantage of it, guys! This is from Arsalan Khan on LinkedIn, Twitter, on X, and he, this is a really good question. He says, what makes governing AI agents different from traditional organizational governance, and how is it implemented?

Saral Jain: It is easy to let agents become extremely powerful, and the risk profile there is then very, very scary. And so for us, we ground ourselves into what we call jobs to be done. And I'll explain that. Every function at Snapchat has a job. Engineering has a job, sales, finance. And the job is for our community, for our partners, for our employees. And we ground ourselves into what are the key jobs to be done for every function in the company.

And then we then decide whether humans are the most capable of doing that job or whether agents can help us meaningfully accelerate that job. And grounding ourselves into that, what is the benefit for our community, our employees, our partners, really helps us focus on the governance aspects as well, because then we can kind of ensure that agents have only the amount of permissions required to help do its job well for the benefit of our community.

Michael Krigsman: And, on this exact topic, we have another related question from, and this is on LinkedIn, and this is from LogiRoot Runtime AI Governance with Verifiable Audit Evidence. Okay, so he's got a good question, and his name is an ad for his company. Okay, fair enough. He says, what type of governance does this system have in place to ensure all agents are acting according to company policy?

Saral Jain: We actually definitely start with an agent registry. We also have inputs and outputs of what an agent should be able to do and what an agent actually does. And most importantly, we have audit trails. We want to be able to understand what happened in production when these agents are running, and those audit trails are extremely helpful.

In addition to that, we obviously have a lot of governance in place in terms of data access, in terms of read versus write of data, in terms of network sandboxing to ensure that agents are scoped to their identities and scoped to their permissions and are only able to do what we expect them to do.

Michael Krigsman: This is kind of an obvious question, and I can anticipate your answer, but how well does that work? You mentioned earlier you don't want agents running amok, but if we look at what happened with OpenAI and Hugging Face? Sometimes agents do run amok despite one's best efforts. So, how do you manage that?

Saral Jain: Our concept is risk tiering. There are certain systems that are not the same as other systems. A minor naming change in the app might be very different than a privacy-sensitive migration that an engineer is trying to do. And so for us, risk tiering is extremely important, and that ultimate human judgment, and autonomy is really important as well. Humans own the final outcome, essentially. So when we think about agents, we are not unleashing agents on one-way doors where the risk is extremely high, where the privacy-sensitive nature is really high.

We are trying to unleash agents on repetitive manual work that engineers might have been doing in the past that agents are much better suited to get done fast with ultimate human accountability. But there's certainly risk at play here. There are no perfect solutions, and we are all learning together. Even from a year back when AI was just a glorified autocomplete to now where engineers are running and orchestrating these massive number of agents, things are moving at such a fast pace that certainly there is risk involved.

But on our side, we have done a good job at trying to mitigate as much of the risk as possible.

Michael Krigsman: How do you control the prompts, or the questions, or the tasks that you're giving the agent so that you don't inadvertently give it a task that then encourages that agent to go off and take a path that would be unacceptable?

Saral Jain: This is where our golden path comes in. When we have these high blast radius, very widely used agents, they are blessed agents that a lot of engineers collaborate together to What we don't want is every engineer and every team trying to duplicate, trying to solve the same hard problem of building a particular agent to do our A/B analysis or to kind of look at some privacy-sensitive migration or to do code reviews in a different distinguished way. We want everybody's energy to go behind these managed agents.

Then we do code reviews for what the prompts of these agents are and ensure that we are learning from each other and putting all of our energy into these blessed golden path agents. So we have a lot of confidence in these. But when an employee builds an agent just for their own use, we also want to let them explore and be creative and not create too much stifling of the innovation there. But the blast radius is much smaller because it's an agent only for an employee.

And all of our agents go through a central MCP gateway, essentially, and a central MCP platform. So we control choke points where we can decide which agents have access to our most critical data, and most sensitive data versus which agents are only prototypes that do not have access to the data but can do a lot more.

Michael Krigsman: So, do you have a classification system, for example, when somebody creates an agent that they classify it as high-risk or low-risk, for example, or other attributes? Yeah.

Saral Jain: 100%, like widely used agents go through severe privacy and security reviews. Our security team classifies the risk tiering as well. But for most employees, if they're building an agent for their own usage that has access to only the data that employee already does, then we do not want to create too much processes in terms of that innovation as well. But for the widely used agents, we absolutely go through a risk clearing process.

Invest in guardrails before building agents

Michael Krigsman: It's really interesting, and I have to imagine that with almost a billion human users per month, if something screws up, you're going to hear about it very quickly.

Saral Jain: Oh, I'll certainly hear about it. I still carry a pager. I go on call when something bad happens. I'll certainly see that page. But at the same time, look, we feel very confident in the value AI is adding to our engineering team, and we are very thoughtful about the risk that comes with it. And ultimately, it's that risk-reward scenario.

We want to put all of the guardrails, and we want more than 50% of our investment actually goes into the guardrails, not in terms of actually building stuff, but putting guardrails in place, whether it's our evals layer, whether it's our code review layer, whether it's our rollback layer, and so on. And that investment actually does not slow us down. It actually helps us move faster because people have confidence in the system now.

And that is, that I would encourage every leader thinking about agentic development at scale is to invest first in the guardrails, not in the building.

Michael Krigsman: Just to elaborate on this point, so you're saying that at least half of your development budget, should we say, on agents goes to the governance and the guardrails, aside from the actual technical functioning of the agent itself?

Saral Jain: Yes, absolutely. I mean, look, the first agent that became popular at Snap was CodePal. It is actually not an agent that helps you write code, it's an agent that helps you review code. It's a guardrail. And so, for us, investing in evals and guardrails and reviews is as important as building agents that can help you do coding faster because ultimately the quality of the software, the craftsmanship really matters when you're building stuff for a billion people.

Michael Krigsman: So, let's go to a question on exactly this topic. And this is from, on LinkedIn, Swami Vaidyanathan. And he asked this about 15 minutes ago. So, he's, I mean, the audience, you guys who are listening, you guys amaze me every time. Okay. And he says this: Are there specific engineering best practice design patterns or guardrails you have developed at Snap that could be emulated in other organizations? And he's talking specifically relating to the AI development lifecycle?

Saral Jain: I think the number one thing I will say is don't solve hard problems over and over again in disconnected ways. Put all of your energy in solving hard problems with these golden path blessed managed agents. So when I say hard problems, it's like if you want to build an agent that can debug crashes of your app, do you really want 1,000 of these agents or do you want one really, really good agent that can debug crashes in your app?

Similarly, if you want a code reviewer, you really should have one code reviewer that can actually really understand your code base. It can understand all of the repositories, it can understand your coding best practices and be to do a good job at code reviews. So it is hard to build these agents. And so doing it, having a central team that is responsible almost as a product to build these agents is really important. Second is please invest in the eval layer.

The agent quality will improve or degrade over time, and you'll never be able to know about it without a very robust eval layer. So investing in evals is really important. And then the last thing I'll say is the governance and the central platforms that enable agents at scale. Things like your MCP platform, things like how to roll back stuff is extremely important as well. So be very thoughtful of investing in these managed solutions by a central team that can own all of the agents is really important as well.

Having said that, the opposite side of it is do not stifle innovation. Some of the best ideas from your team will come from people who are experiencing the problem. So there are two sides to this, there is the blessed agents but there is also let each engineer innovate essentially in a constrained way so that you can get the best ideas to the top.

Michael Krigsman: You said that building agents is hard. Is it any more difficult than traditional software development, or what did you have in mind? Why is building agents hard?

Saral Jain: Yeah, one of my favorite things to say is building prototypes is easy. Anybody using AI can build a demo app or a prototype in all almost no time, but building something that is production-scale, safe, secure, that can be used by a billion people is hard. I wouldn't say it's harder or less hard than traditional software engineering, but I would say it's hard.

And I think in traditional software engineers, there are teams that own software, there are teams that own all of the inputs and outputs to the software, they own the software development cycle, they own it as a product. And I think agents are similar, you need to own them as products so that you can improve them over time, you can understand when they regress, and you can kind of understand the quality of the software that is being generated.

Michael Krigsman: Now would be an excellent time to subscribe to the CXOTalk newsletter. So, go to CXOTalk.com and subscribe to our newsletter so we can invite you to more of these events. All right. This is another great question from Supriya Desai, who says, if you were guiding an organization new to using a system of agents, and they only had the capacity to focus on three things, what would be the most, three most important principles you'd say they should follow for, you know, guardrails, risk, timing, talent, based on what you've learned on Snap's journey?

Saral Jain: The first thing I will say is don't think about building 50 agents, think about solving one problem. I think that is the key. If you have an existing problem that your team, your organization is facing, think about that workflow. Think about whether AI can help you accelerate that workflow, and AI should not be bolted at the end. You need to redesign the workflow with AI in mind. Most people jump to that, the number of agents or just having an agent first, but you have to always start with the problem.

So that would be my number one thing. The number two thing I will say is context beats model 100 out of 100 times. There is an obsession about what is the best model right now, and it changes every three weeks anyway. We have realized that almost all models are really good at catching bugs now.

So the problem is never what model you used, it's really more about the context you had given to your agents, like whether they understand your information, your systems, your code, and I think investing in that context is really important. And then the third thing I will say is what I've said before, which is invest in evals, invest in guardrails, because that will help you when things regress.

Output metrics and the coordination tax

Michael Krigsman: Folks, when it comes to context, just so you know, next week, a week from today, we have as our guest the Chief Product and AI Officer of Atlassian, who's going to talk a lot about context. And, we also have coming up the Chief Product and AI Officer, I don't think that's her exact title, from Salesforce. So, again, we'll be talking about these issues, so stay tuned for lots of discussions on these topics. So, Saral, how do you measure the value or the benefit of agents?

Saral Jain: I think it's very well known in the industry that measuring output or productivity of engineering is extremely hard, it's a very, very hard problem to solve and we don't try to obsess about activity metrics, things like number of PRs or number of tokens. I think these are all gameable input metrics. What we ultimately care about in terms of the value is the output. Are we shipping better software faster? That is the most important thing to me as an engineering leader and are we improving on costs?

Are we having less number of outages? Are we improving our meaningful top line of the business? I think those output metrics are ultimately what we ground ourselves to, whether it's humans or agents, to see if we are seeing the value.

And the interesting thing is with agents, we are seeing a lot of movement in some of these output metrics, essentially, whether it's identifying cost opportunities or identifying how can we move from an IDR to a production A/B in a much smaller amount of time than the weeks it used to take before. That gives me a lot of confidence that these agents are helping.

Michael Krigsman: You've spoken a lot about, essentially, efficiency metrics, doing things faster, having higher quality code. Do agents help you do something new, different? From an innovation standpoint, can you talk about agents as the support of innovation?

Saral Jain: I will give an example. So we have hundreds of repositories over thousands of services essentially, and sometimes it's really hard for us to make a change that is cross-cutting across all repositories of our code bases. And these could be performance improvements, these could be cost savings, it could be making things efficient. But the coordination tax between teams is too high.

Like if you are going to make a change that is going to change 20 services, then you will need to talk to owners of those 20 services, convince them to make the change, get it above the line of their roadmap, and then kind of go through that entire process. And sometimes that coordination tax is too much. Whereas with agents, we can unleash an agent on this problem. It can generate 20 PRs. It can give context about what this particular change is.

It can test it, it can ensure that it gets reviewed by another AI agent, and all these 20 teams need to do now is look at the PR and ship it in a safe way. And that coordination tax is going lower. What I always say is if the cost of building is cheaper than the cost of having the meetings, then have fewer meetings. I think that's really what we're seeing right now with these agents, and that's why we're improving the performance and cost efficiency.

Context you can give, judgment you cannot

Michael Krigsman: Okay. And then, Supriya Desai says, what's hardest about managing agents that's different from managing people?

Saral Jain: They both have their challenges, but managing agents certainly comes with their own challenges as well. Certainly the guardrails that I've talked about quite a few times help us a lot. Our worst-case scenario is an agent does something that it was not supposed to do. It touches a production surface it was not supposed to do, it touches a piece of data, it modifies a piece of data it was not supposed to.

So that is really hard, is humans come with an extreme amount of judgment, right, like an agent you have to teach the judgment and I think that is where a lot of our energy goes is the safe thing to do, the risk-averse thing to do.

Michael Krigsman: How do you teach that judgment? I often think of agents as being the most brilliant 10-year-old in the world.

Saral Jain: You don't teach agent judgment, you just put the guardrails in place so that the agents just cannot do what you don't want it to do essentially. And of course there is a lot of prompt engineering that goes behind the scenes as well to make sure that they're inherently safe. But as we have found out very, very publicly in the last few months, prompt engineering by itself is not enough. You need systems in place to kind of prevent the bad things to happen.

Michael Krigsman: Can you talk about that context that you mentioned earlier? So, at Snap, what does context mean? This term is being used so often right now, and yet, it's a kind of vague term.

Saral Jain: To me, context is the information that has existed at Snap for the last 15, 16 years. The company has existed for 15 years, and we have so much institutional knowledge that has been preserved over documents, over code bases, over meeting notes. And if you can give your AI agents and models access to some of this context, we have seen that they far outperform the base agents essentially, and so that is why investing in the context layer and giving it information about your company is really important, not just having the base agent.

Michael Krigsman: But how do you make the decision of what information is needed so you're not giving it too much, but at the same time, you're giving it sufficient information to accomplish whatever needs to be done?

Saral Jain: That to me goes back to this concept of jobs to be done. We start with what problem, what business problem are we trying to solve? Who will it benefit? Who's the ultimate consumer here? Once you start with that, you understand what access of information is required for a human or an agent is immaterial. What access of information is required to solve this problem? And then you give it, the agent, access to that information.

So I always tell people, but ground yourself, please ground yourself to the problem you're trying to solve and that will help you encapsulate what information, what context, what data is required for either the human or the agent.

Michael Krigsman: Can you give us an example of what you're talking about? Something concrete in relation to Snap to help us really understand the nuances of it.

Saral Jain: Right, a good example is CodePal which is our code review agent. In the past what people used to do with code reviews with agents is when a piece of code gets submitted as a code review, the agent has access to that piece of code review. It can find the bugs in that code review, but that code review only contains the diff of what existed before and what new code was being generated. We realize that is not enough.

Our CodePal actually has context of not just the piece of code that is changing, but literally all of the surrounding code that is not changing as well. That gives it the ability to understand coding best practices for this repository. It helps you understand what are certain assumptions that the engineers were making outside of the code that was being touched so that it is able to identify much bigger surface area of bugs in the code because it has all of this other context that is not part of the PR.

So that hopefully is a good example of why context matters. But at the same time, we do not give our CodePal access to our Oracle financial data, because it does not need that data. And so being very thoughtful in terms of what access a particular agent needs to do its job well is really important because when you think about engineers, they're not looking at code review diffs only in context of the diff.

They already know context about your coding best practices of the company and all of the context outside of that diff as well. And so we want to make sure our CodePal has that as well.

Michael Krigsman: So, that's really one of the differences between managing people and agents. And you alluded to this earlier, is people have that context built in, coupled with judgment, and agents simply do not.

Saral Jain: Exactly. I think that's a really good summarization.

The risk is unowned code

Michael Krigsman: Again, you guys should be asking questions. When will you have the opportunity to ask Saral Jain from Snap, you know, whatever you want? So, take advantage of it! Use this opportunity! Okay. This is from Lisbeth Shaw, who says, You spoke about AI agents reviewing AI-generated code, but humans have the final approval. With the speed and scale that AI agents run, won't humans get overwhelmed?

Saral Jain: The sheer amount of code being generated by agents is really hard for humans to review every single character, every single line of that code. What I'm saying is, before a human even gets to it, we need to use AI and we need to use these AI guardrails to improve the quality of the code that AI generated. But ultimately, humans will still take the final decision.

And I'm not saying that they need to look at every single character of the code, but they do need to understand the spec, the outcome of, and what the system is going to do as a change from this. And in addition to this, there are risk tiers. An internal dashboard that is used by a few employees that does not have access to sensitive data, we can unleash AI agents and it can actually be 100% AI-generated with very little human reviews, maybe even auto-approved in some cases.

But when a change is privacy-sensitive, when it goes out to a billion people, we do need to make sure humans are ultimately accountable understand the spec, understand the code, and are able to vouch for the outcomes of this code.

Michael Krigsman: Maybe drill down a little bit into these two use cases so that we have a clearer understanding of how this actually works. So, say the one use case where you have agent code that is creating an internal dashboard, okay? And the other use case, you have something having to do with security or privacy that will directly affect your users. Take us down these two paths so to compare them.

Saral Jain: In the first case where you have an internal dashboard, what we're seeing is a lot of these internal dashboards, especially from teams that are even outside engineering that were previously bottlenecked on the engineering team to build the dashboards, are just building these completely AI-generated dashboards with a lot of guardrails that we have already put in place. Things like code reviewer, things like the MCP platform ensure that these agents or websites or internal dashboards do not do things that we don't want it to do.

It does not have access to data we don't want it to have access to. But the code is still very much auto-generated, and the bar of a human reviewing every line of code or auto-approving is much, much lower for these internal dashboards. Versus a production system where it is maybe a security risk or whether it's a privacy change, that's a completely different risk classification essentially where we absolutely do expect humans to understand, review, and ultimately own the final output essentially including all of the code.

Michael Krigsman: And when you say ultimately own, I don't want to put words in your mouth, but are you ensuring, as part of your process, that the human is going through every line of code to make sure that they fully understand it in those sensitive areas?

Saral Jain: Yes, 100%, absolutely. When a PR gets generated, AI does the first pass. It gives a thumbs up to whether the AI thinks whether this code is safe, but a human ultimately looks at that change, especially if it's a very sensitive change, and goes through the PR and ultimately says yes, that this is ready to be shipped. And then of course after that we have a bunch of guardrails as well in terms of testing and rollbacks to ensure that we limit any bad blast radius from these changes. But humans are ultimately accountable for any privacy-sensitive change.

Michael Krigsman: So, from that standpoint, I assume there is, at the end of this process, no difference between the AI writing code I'm talking about the sensitive code, the AI writing code versus a person writing code, because in both cases, there are equal levels of accountability and expectations of review. I'm not trying to put words in your mouth.

Saral Jain: Yes, no, 100%. I think the thing I always say is the risk is not, and we have to understand this, the risk is not AI-generated code or human-generated code. The risk is unowned code, and that's what we're

Michael Krigsman: What does that mean for you at Snap? I'm drilling down into this because it's so important.

Saral Jain: Unowned code is what gives me worry because people do not vouch for understanding every aspect of what a system does. Look, software engineering is really important from a craftsmanship perspective for you to understand the inputs that go into the system and the outputs you expect from a system. And if a human or an AI ultimately does not own that code, then it's really hard for us to be able to ship it to a billion people. And so that's why I say the ownership is the most important thing.

The accountability is the most important thing. Whether it was drafted by AI or human is immaterial. It's the ultimate ownership. And what I do say is the ownership and the judgment is where the humans are ultimately the accountable owners.

Michael Krigsman: And obviously, in less important, you know, less critical code, the degree of scrutiny will be significantly less, commensurate with its lack of risk profile or capability.

Saral Jain: Ultimately, we want to be able to move faster for these areas where the blast radius is much smaller. And so it is, even before AI, there was a difference between the different risk classifications of systems. And so the same concept still applies. And in these lower-risk systems, I think we can use AI a lot more to check AI's work.

So it's not to say that you should just generate AI slop code, you should still have all of these systems in place to be able to review the code that was generated using AI, to be able to have fast rollbacks, to be able to put evals in place so that you can understand regressions in the system really quickly. But you can rely a lot more on AI to do all of this versus that ultimate human judgment that is reserved for the most critical pieces.

Working with unpredictable AI systems

Michael Krigsman: Folks, for a deep dive into some of these agentic security issues, we recently had as a guest Anand Oswal, who's the Executive Vice President of Network Security Products at Palo Alto Networks. So, go to CXOTalk.com and just search for Palo Alto, and he discusses some of these identical issues relating to visibility and taking an inventory of your agents, and so forth. All right. We have an interesting question from Andy Grey on LinkedIn, who says, AI is not very predictable.

To assume we will know how it all pans out over the next 12 months is a stretch. AI can be a little bit naughty. So, to generalize his question, how do you manage this probabilistic aspect of AI development?

Saral Jain: The core of this question is we are going from a deterministic world to a non-deterministic world because the inputs and outputs are different with AI. But to me it depends on where you're exposing AI to your ultimate audience. If you're putting a chatbot in front of your audience you have to do an extreme amount of work to ensure it has all of the safety and the guardrails in place to make sure it kind of is doing what you expect it to do.

If you're using AI for engineering internally in a company, then you have to think about it very differently. And I completely agree with the premise of the question that 12 months is an eternity in terms of AI software engineering. 12 months back, we were using AI as a glorified autocomplete. Even since then, now we are orchestrating all of these hundreds of agents essentially. So things are moving at such a fast pace, that there is inherent unpredictability and risk, as the question implies.

But that is where our investment in the guardrails, in the evals, in the code reviewers come in handy, because it helps us be on track when things change. But the risk businesses are taking today, or the risk businesses have to take today, is because of the benefits that can come in terms of being able to much, much quicker using AI.

Agents beyond engineering, and human skills

Michael Krigsman: This is from Arsalan Khan on Twitter. He says, beyond engineering, are there other departments, finance, sales, marketing, and so forth, in the organization that can easily use AI agents for their own needs? And here's the kicker: How do you manage this possible spaghetti nightmare?

Saral Jain: The answer is absolutely yes. We have found so much benefit in using AI agents, not just for engineering, but for other functions within the company as well. A great example is customer support. Snapchat community often reaches out to Snapchat for various advice or help around using Snapchat. And we have seen that we deflect more than 3.5 million questions every single month using AI.

So that is a great use case where AI agents can help us focus on solving hard problems for our community and deflect some of the more often asked questions that AI can. Similarly, we are using AI agents for a lot of other purposes like our trust and safety team, our sales team efficiency, our finance and people team transformations, and so on. So the premise of the question is absolutely correct. We should use AI for building agents across all of these different functions.

Michael Krigsman: What does this do to talent? Talent inside engineering, talent outside? What is the impact of agents? And very quickly, please.

Saral Jain: The things that mattered before are still the things that matter, like things like curiosity, being able to learn new stuff, having the software development fundamentals still are the things that we screen for and the things we care about essentially. And so that has not changed for us.

Michael Krigsman: If you had a son or a daughter who was in high school looking at careers, would you advise them to become a software developer in this world?

Saral Jain: I think software development as a profession is going to change quite substantially. I would advise them to kind of pick a career where they still learn how to learn. I think that to me is the most important skill at this point, is the natural curiosity, being able to learn, being able to be very good with using AI is going to be fundamental. But as a career profession, I think AI will have an impact to a lot of careers, including software development.

But the core skill sets of good software development best practices and the judgment is absolutely going to matter going forward as well.

Michael Krigsman: What advice do you have for more junior software developers who are seeing agents take over the writing of code they would otherwise be doing?

Saral Jain: I think the value humans add is the judgment. AI can be confidently wrong in very subtle ways and that is where the software engineering best practices, the judgment, the expertise, the experience comes in handy. And so we're not anywhere close to AI being able to replace that judgment essentially. And that is the advice I would give people is focus on the basics, focus on the software development best practices.

Michael Krigsman: So, developing judgment, the personality traits you just described: judgment, curiosity.

Saral Jain: Curiosity. Yes, exactly.

Token spend and model routing

Michael Krigsman: Another question now from Chris Petersen. He says many companies have had to recalibrate their tokenomics spends lately because of both base cost and spend volatility. Are these worries for Snap as you roll out more agentic services?

Saral Jain: These are absolutely something we are thinking about. We have our own internal dashboard that we have developed where engineers have access to their own token consumption so that they can understand how they're using tokens. And in addition to that, there's also an emergence of really, really powerful open-source models in addition to these powerful frontier models. And we have model routing that we are developing that can help us use the right models for the right tasks at the right time.

And so token spend management is going to be key for not just Snap but for all organizations going forward, and we are definitely investing a lot in that area.

Michael Krigsman: Did you go through a token-maxing phase where the goal was to use as many tokens as you could?

Saral Jain: You know, one of the best decisions we took is we never created these vanity dashboards where people were projecting the amount of tokens they used because, as I mentioned, these are input and activity metrics and they have absolutely no correlation with your output and the business impact. And so we did not have that token maxing phase as is being talked about in the industry, but we certainly are being very thoughtful in terms of how we use tokens going forward.

Michael Krigsman: Can you talk briefly a little bit more about open source and open weight models, what you're doing, what the impact is, what the challenges are?

Saral Jain: These models are getting really good very quickly and we are more at the exploratory phase right now. We use frontier models from the big labs. We also experiment with open-source models. And our big challenge right now is deciding dynamically when to use what so that that burden is not on individual engineers and certainly non-engineers. Because as I'm sure everybody can attest to, by default, everybody pins their usage to the most capable model all the time. And that is not a very efficient thing to do.

So for us, open-source models are the right counterbalance to it when you can use it for the right. Great, thanks.

Advice for leaders and the risks ahead

Michael Krigsman: And folks, if you're interested in open-source and open-weight models, we recently had the chief technology officer of Mozilla, you know, that makes Firefox, as a guest. And he's coming back in October specifically to talk about open source and open weight models. So, if you care about this topic, you definitely want to check that out. Saral, you are also the Chief Information Officer of Snap. What advice do you have for CIOs at other companies who are looking at organizations like Snap and your use of agents?

But, of course, you are a pure technology company and obviously very innovative in this. So, what advice do you have for CIOs who want to get more deeply involved with agents?

Saral Jain: Four pieces of advice very quickly. Number one, start with a problem, not with an agent. Number two, focus on context, not on the model. Third is focus on evals and guardrails. And fourth, and maybe the most important thing, is start at the top. You cannot ask thousands of people to change the way they work without doing it yourself. So it starts with the leaders. Leaders need to embrace AI.

Michael Krigsman: Are you talking about senior technology leaders or senior business leaders? How much does the average businessperson or senior business leader need to know about development with AI? Agents, for example.

Saral Jain: In my view, it should be all leaders, irrespective of whether they are technology leaders or business leaders, need to be AI savvy. AI is changing the way we work across not just engineering, where we have seen the most profound impact, but also other functions at Snap, like the way we think about sales efficiency, the way we think about our people team, the finance workflows is changing with AI, and it starts at the top. So I would encourage all leaders, including CIOs, including CFOs, to think about how they can use AI to benefit their organizations.

Michael Krigsman: We've had other folks on this show say the same thing. You need to understand what this is. You don't have to become an expert coder, but you need to understand it. Saral, very quickly, what worries you most when it comes to AI agents? At 3:00 in the morning, you wake up and you're thinking about something that worries you when it comes to all of this. What is that?

Saral Jain: AI agents can be wrong in not obvious ways. They can be wrong in very subtle ways, and they can be wrong very confidently. And so, the biggest worry I have is ensuring that the guardrails are put in place such that we can detect when something is bad. And the second is a change of culture. I talked a little bit about that ownership mentality and owning the code, essentially.

I worry that over a period of time it would be easy for people to ship changes they do not understand, and that cultural craftsmanship tenet is really important to me that you ultimately are the owner of your code.

Michael Krigsman: We have just a last question sort of coming in under the wire here on LinkedIn. This is from Syeda Zeenath, who says, what safeguards exist to prevent agents from introducing security vulnerabilities or technical debt?

Saral Jain: We actually use AI to improve our security posture, and it has been a big, big kind of beneficiary for us. But at the same time, adversaries have the same access to the same tools and very powerful models. And so this is something we are thinking a lot about. This is where the guardrails come in place. Every single change still has to go through privacy reviews, security reviews, ensuring that our best practices are not let go, essentially, is really important for us.

Michael Krigsman: So, you are putting every line of important code through those important security reviews.

Saral Jain: 100%, yes, absolutely.

Michael Krigsman: But still, things can, you know, problems can sneak in.

Saral Jain: And I think we are seeing that the amount of security vulnerabilities, vulnerabilities not just for Snap but across the industry has exploded in the last year. And we are seeing that in bug bounty programs, we are seeing that across the industry, and it is an unfortunate reality. When more code gets generated, more vulnerabilities will be present. And that's something that our security team takes really seriously, and we are working to invest a lot in that area using AI.

Michael Krigsman: So, I'm assuming you're not allowing your developers or your agents to bring in MCP servers that they found on GitHub or that their friend made.

Saral Jain: Yes, 100%. That is a very good assumption. I think we have this managed MCP platform and we very much protect what data and what systems agents have access to. And we actually worry a lot about third-party vulnerabilities as well because your supply chain can introduce very complicated security vulnerabilities and dependencies as well. So we are being very thoughtful about that.

Michael Krigsman: In one sentence, what's next for AI agents at Snap?

Saral Jain: We will definitely see a lot of long-running complex agents going forward. We will also see more auto-approvals where the risk is lower. And generally, I think we're in very early days right now for agentic software development. So more to come in the next few months.

Michael Krigsman: Okay, Saral Jain is Head of Engineering and CIO at Snap. Saral, thank you so much for taking your time to be with us. I'm very grateful to

Saral Jain: Thank you for having me.

Michael Krigsman: Everybody, thank you for watching! Now, before you go, check out CXOTalk.com, subscribe to our newsletter, and we'll see you again next time. Have a great day, everybody!