Rogue AI? How S&P Global Stops It Cold!

Uh just a little overview of S&P Global for those of you that don't know us. Uh we are a financial information and analytics company. We provide market intelligence, credit ratings, benchmarks, uh insights for governments, corporations, and investors worldwide. We're particularly excited about AI uh primarily because one of our core mission statements is around how do we amplify essential intelligence for our customers. Uh we think that uh our internal capabilities, our enterprise capabilities capabilities along with the new Agentic solutions that we're incorporating into our products uh provides a real opportunity for us as a business.

We're applying AI in two fundamental ways. Uh the first one is for our customers. So, we're looking outward. We're incorporating Agentic solutions into our products that our customers are leveraging on a daily basis. We're [clears throat] also looking internally. So, we're deploying Agentic solutions across the enterprise that are helping our workforce improve their productivity and get overall capacity expansion.

We partnered closely with AWS along this journey and we're using many different products uh from their stack. And one of the core ones that we've really found a lot of value in is Bedrock Agent Core. It's really allowing us to build solutions and what used to take uh months or weeks, we're now able to do it in days and hours. And so, we're going to talk a little bit today about how we think about this from the standpoint of how do we leverage Bedrock Agent Core to deploy these solutions with safety in mind. Um so the agenda today, we'll talk a little bit about the challenges that we faced as we started to scale up Agentic solutions. We'll introduce a Agentic reference architecture that we use to allow our developers to build in a common way. And then we'll walk through our agent safety control as well as our FinOps circuit breaker. And then talk through results and key benefits and hopefully have enough time at the end for Q&A.

So as we start down this path like most of you, there was a number of challenges that we faced immediately and then also projected as we thought about what this would look like as we really started to scale agents across our workforce. We operate in a regulated business like most of you. Uh and so thinking about how this is going to impact our ability to remain compliant uh was top of mind. We've learned our lessons through the cloud. So we've all had the uh rogue query or script all of a sudden drive up our cloud costs for a day. So we definitely anticipated that by deploying a number of agents, we're going to have unpredictable costs. And then we realized that humans are now the bottleneck. So as we start to deploy agents within the workforce and incorporate human in the loop, we recognize that we're going to eventually have to deal with that bottleneck. So those were the challenges that we faced. And one of the first questions that we wanted to answer was how do we get all of our developers rowing in the same direction and building common capabilities across the enterprise?

And so we developed an Agentic reference architecture. This is important because it not only allowed our developers to to deploy solutions in a common way, but it also allowed them to really think about a holistic solution as they were going through the initial design of these new capabilities. So not just thinking about the model that they were going to use or the prompt that they were going to develop, but thinking more around what's the observability? What's your FinOps metrics? How are you going to observe and monitor monitor these solutions in production? So, we went from everything from core design thinking to core engineering, such as tooling, memory management, agent UIs. We started to establish standards for how agents are going to communicate and how humans are going to interact with those agents. We started to think through resiliency architectures, including how are we going to incorporate evaluations, security, and scaling. And we also thought about the ecosystem as a whole. How are we going to deploy registries? How are we going to give our users the ability to discover these solutions? So, this allowed us to present a common framework for our developers, get them all rowing in the same direction. But, the next question that we wanted to

ask is what happens when the unexpected occurs? We know these agents are going to have some unexpected behaviors. How do we deploy a solution that allows us to truly scale these agents up with a lot of confidence? As an organization, like all of you, we want to move fast. But, we can only do so if we have real confidence that the agents that we're deploying are not going to go rogue and we're going to be able to control those those agents with the necessary guardrails in place. And so, the question that really came up for us was we need a pattern that we could use to detect anomalies not from across a number of different sources. So, not one source, but many sources. >> [snorts] >> [snorts] >> We needed humans to remain in control. So, we wanted that human in the loop. And we also wanted to provide our developers and our operators with both a surgical capability where they could go in and isolate an individual session, but then also a much broader capability where they can shut down an entire organization if they needed to. So, this was the challenge that we had, and we realized at that time that we really need an agent safety control panel. So, I'm going to invite Jeff up here to

walk through the solution that we built. >> Hey, good afternoon, everybody. Thanks for spending some time with us. So, as Todd laid out, that's kind of a simple diagram in terms of really what we needed to solve. But before we get there, let's talk a little bit about how did we land on AWS? So, like all things, start out looking at your vendors. We talked about three. We ended up POC'ing two. When you start going through that whole reference architecture that we had built, there's a lot of criteria that go into trying to make sure that your tooling, your vendors are going to meet your needs, especially in a space that's changing like hourly. I mean, how many people here have had that mail come through that says, "Have you seen this new thing?" When was it released? Oh, yesterday at 3:00 kind of deal that everybody's running at. And so, And so, when you talk about AWS for us, it's managed infrastructure. We've got the ability to have scaling, availability, patching, whole bunches of things that you just get by having the platform. You get cost efficiency, basically per invocation serverless at the point of using Bedrock agent core. Security, the isolation of the actual VMs under the covers basically sets up some very good security posture to start with, and then that leads into things like IAM, secrets manager. All of these things that are currently baked into the AWS ecosystem that we use on a day-to-day basis. It means we didn't have to actually create a lot of this from scratch. We could lean on a lot of the institutional knowledge that we built from our cloud migrations. When you start talking about model access, basically, you have inherent access to Bedrock at this point. Again, you get this out of the box. So, we took all of these things in we sat down. We sat down with the reference architecture. And then you need to figure out

where your pain points are going to be. As Todd pointed out, things can go off the rails. Things are non-deterministic. We kind of signed up for that in the agentic world. And so, when you take a piece like a very simple diagram here with the conversational agent, basically, the agent registers out with Bedrock. Control Panel picks that up in terms of we now have visibility to everything that's running in the environment. So, once you're registered with Control Panel, we have the ability to list out what's there. You have the ability to talk about this in a little bit. See what's under access control. We have the ability to do session management in terms of terminating sessions at scale. And the ability for the agent to actually register. It's a pretty straightforward architecture when you're talking about what it actually does. The second hardest piece is admitting what you don't know. So, Control Panel is modular. It's built on a set of services such that we can actually pull things out and put things in as needed. So, as we're looking through what Control Panel can actually do, the second question inside the framework is where does it get deployed? We're a big company. We have small applications. We have big applications. We have divisions. We have business lines. So, it needs to be flexible enough to be deployed at a single VPC with a single application with a single agentic infrastructure behind it. All the way up to you need to be able to aggregate through a business line. Such as you can see everything that's going on inside a business line. And so, the ability to be able to take this and have it applicable at all of those levels is also a very interesting undertaking a problem that you need to solve at scale when you're talking about this. One of the reasons why we started small with session management and just a very particular piece. Fundamental A that you needed as part of a safety control, but you also don't end up boiling the ocean. There's a lot you can do with this. If you try to solve all of those problems at once, things get very complicated very quickly. As an example, one of the things you talk about in times of like how we built this in the thought process that went in. You have by session, by user, by agent, by customer. Got to be able to control it all at those levels. One of the things you are interested in when you're building your data model at the back end, if you flop customer and agent in terms of sort of that back end modeling that you're doing, you start running into problems when you have customers that are coming in that are using multiple agents across multiple applications. You sort of lose the visibility there. And so, when you're out there building your own systems, you have to look at it from that level. You have to be able to be able to look at the individual applications and make a decision about where you need to intervene, where you need to be able to get involved in those systems.

This basically then gave us how do you control blast radius? control blast radius? So, as we walk through that model, we start looking at the ability to say by session, by user, by customer, by agent. Really, we looked at those in terms of severity. So, by session, this is a scalpel. It could be an observability problem. It can be something is just off the rails, but I need the ability to control just that fine minute thing. With a single session inside a single agent being off the rails, I don't want to take a customer base down. I don't want to take an entire agentic application down. However, in a non-deterministic world, if you've got the right hat on, you have to be worried about what happens if an entire agent is off the line. So, as we move forward here, the control panel with human in the loop gives you the ability to at each of these levels terminate sessions, users, customers, and agents. Today, this is very human in the loop. The way the APIs are currently laid out, as we mature and we get better, we're going to open these up to our observability systems and allow this to get to be much more programmatic. One of the things about agentic systems is they grow very quickly, if you're successful. And the ability to say humans are going to be involved in all of these transactions, probably isn't a safe way of thinking about things. Things can go off the rails very quickly, and sometimes having the computer make the decision to start those down, the thing that's nice about the way control panel is built, that's going to grow with our maturity. We've got the ability to grow into that. So, humans are involved today, as we learn lessons, as we learn look at the observability metrics, what those are really going to look like in the wild, we've got the ability to build on top of this.

things go off the rails very quickly. So, I have all of these systems, is there a way or is there something that is very deterministic that you can use as a gate to potentially say, "I need this to stop now." And for us, finances seem like a really good place to go make that decision. So, at S&P, we actually have cloud budgets in a CMDB. It's actually an item that we maintain as part of our FinOps program. And so, in the agent space, we decided to potentially use those to control the out-of-control spend. So, we built out a part control panel application that will pull these budgets down into the individual accounts, and we populate what's called a circuit breaker. And what circuit breaker does is looks at the actual spend against what your budgeted spend is for a given month. What this allows us to do, this is not a dollars and cents. This isn't necessarily meant to be part of our FinOps program. This is dealing with big triggers. 1x, 2x, 3x of your monthly budget, where you know something is wrong. The way this currently works is you can feed it an arbitrary set of AWS services. If you trip the circuit breaker, we will take those services offline. Effectively, we will apply an IAM policy that will just basically blanket across the board deny those services from coming up. Two reasons why this is good. One, we don't touch any of the running systems. Everything is still in place, they just can't do anything at this point. This allows your operations teams to get involved, this allows other parties to get involved, you can investigate. If for some reason the breaker fired for a bad query, something that's not a problem, getting it back online is fairly quick. We have the ability to set gates in this, which is I can go with approvals, reset my monthly budget, but it'll go back to what it was the following month. So, there's an exception process built into this. So, between circuit breaker and between

control panel, it is defense in depth. It is very much the idea of as you are looking at your agentic systems, as you are going through and at where these apply. If you're not starting with that reference architecture, it gets very hard to figure out where to apply these. These are not blanket rules. These are rules that are going to need to be applied inside your systems. And so, one of the big pieces that I hope everyone takes away from this is as you're looking at this, there is a ton of planning that goes up to the front side of this. So, this leads us to we have control panel. So, near real time, it's multi-signal in that we take fin outs, we've got observability data we can feed into this. We've got the ability to um really allow teams to be flexible while providing them a paved road to move forward.

Next slide, we're going to look at a demo. So, it'll run real quick, 2 minutes. One of the things I'd like for you to take away though is as we go through this, it's just a conversational UI. We're going to first add this to a control list. So, in the session management, we aren't just terminating sessions. We have the ability to actually access control because of the way Bedrock agent core sessions work. So, we have the ability to actually prevent it from working from the front end UI. Very similar to circuit breaker, if it's not really a problem, we actually allow you to go back to your session in the exact state you left it. So, the other piece of this is when we do go through and we see it at the terminate session, you will notice that we also add it to the control list. This is important because if you actually end up with a session ID and effectively a memory ID on the back end, you can still rehydrate a session. And so, in the model we took, you can terminate it. If for any reason you still have that data, you still can't get back on the system. So, at this point, standard conversational UI. Then what we're going to do is we're going to grab the session ID. You've got the ability to filter directly for what you need. This is going to pop us down into the correct agent and the session we're currently working on. And the first thing I'm going to do is I'm going to add this to our control list. So, this will basically give a reason. Once you click okay, at this point, that session's off the wire. And you can do this at the session level, the user level, the customer level, and the agent level. So, when I go back in at this point, that session's off the wire. Friendly message back, left up to the development team for what they want to do. But as you go back through, you find out nothing's wrong. One click, say yes. The session's now back on the wire. As we go back through, you'll be able to Just refresh the page. Notice everything from our previous session is still there. So, now we're going to go through, take the long way around in terms of being able to just click through the agent, see the drop-down list so you can see what an operator would see if they didn't have the session ID. And at this point, we're going to go and terminate that session. At this point that session is off the wire. So, you'll notice that it's added to the control list. Also, full audibility. Also, full audibility. There's logging behind all of this, so every action the control panel takes, we have the ability to bring back in. So, if for any reason you need this for legal purposes, everything is tracked in what's currently going on. From here, you go back into the UI, and at this point that session is now permanently off the wire. So, with that, thank you for your time. I think that's where we're going to end.