Unlock AI's Power: Slash Risk, Boost Public Services!

Thanks to y'all for coming. Um, I'm going to talk a little bit about reducing risk while improving services and operations in the public sector. So, I'll just get started by introducing myself. Um, I'm Rebecca Nissan. I'm a delivery solutions architect or engineer as part of our uh healthcare and life sciences vertical and so I mostly spend my time advising healthcare payers and providers who work with data bricks on how to best leverage our our platform but before I did that I actually worked as a behavioral scientist and did a lot of work in human- centered AI as well as responsible and trustworthy AI so hopefully you'll see some of that lens in the next couple So if uh if I'm coming from the healthc care lens, you know, why does this necessarily relate to public sector and why do what do we have in common here? I think uh one of the big things is that

we're up against a very similar challenge. When we build AI tools, we're accountable and visible to the public, which is exactly how it should be. But when we are building AI, like we're not uh making restaurant recommendations, right? We're like deciding people's futures. We're deciding who gets resources. We're diagnosing patients. So, that's a pretty big deal. And we really can't mess it up. And so, we we do have to innovate though because there's harm by a mission, right? If we don't bring the best of technology to the public sector, then we're actually harming people by not by not trying to innovate. So, the question that I'd pose is how can we innovate with AI while sufficiently mitigating the risk? So I'm going to try to talk about how we can do that with data bricks. So as I said I always think of things through kind of a human behavior lens and I think that good AI needs to be both trustworthy and trusted. So from a developer perspective so those of us who build AI tools, we need to ask is my AI tool trustworthy? Is it good enough to actually put out into the world? We also have to think though about end users because if we build AI tools but then people don't trust or use them then we haven't really done anything assuming of course a use case where people are supposed to be the final decision makers as in healthcare as well as in various parts of the public sector that is often the case. So as developers we need to ask at throughout the process of developing and deploying AI we ask all these questions at the beginning when we're prototyping we say you know is there any obvious hallucination to my naked eye does this seem like a reasonable chatbot do the LLM judges deem it safe but that's not really good enough that's just more AI I mean that's helpful but it's only a start do we have good metrics in terms of the typical kind of accuracy metrics that we would use for this particular type of tool or model and then do we actually have positive end user feedback and have we pressure tested this with the people who are going to be using it and when end users evaluate whether they trust an AI tool they have similar questions you know is there anything obvious here that seems off does it give an answer that's different than what I would decide and what I would expect are there mistakes with dire consequences like does this tool tell me as a clinician for example example to make a diagnosis that I totally disagree with that I think would really harm the patient. That's going to be a red flag. I'm not going to trust that tool. And also, do I know what's happening under the hood? Is is this something where I can actually check its work? If I'm going to make a high stakes decision, I want to know what exactly I'm basing it off of. So, I want to start with the developer

perspective and talk about how we build trustworthy AI. And I actually think it's pretty simple. So so much so that you could put it in one slide. Uh which is really that on in data bricks we think about evaluationdriven AI development which means that in order to build trustworthy AI systems we iterate and do evaluation at every single step of the process. So imagine you have a use case where you're trying to build an AI tool. You've prepared your data. You've done all of that important data engineering work. you've created some tools or a vector index or essentially given an AI agent some context to actually be productive as a tool. And then as you're prototyping, you have to actually look with your own eyes and see does this seem like a reasonable tool. That's the very very first and most elementary step of of evaluation. Then we can vibe check the quality with LLM judges. That's built into our AI playground during that initial evaluation phase. that's also built into our agent evaluation uh framework which I'm going to talk about and there are built-in LLM judges so you don't have to actually go bring those in. you can just uh use MLflow evaluate and you'll get LLM judges that are giving you that initial take on whether your prototype looks good but that doesn't scale right so I'm looking at a couple of examples I'm like okay this seems like a good answer this doesn't seem like a good answer but I need to test this on a lot a lot of examples and I don't always have that many examples so we actually offer the ability to create synthetic data which can be really helpful um of course it's still not good you then need to go and iterally iteratively identify and fix the root causes of whatever the quality issues are that you're seeing. Um, and most importantly actually collect feedback from stakeholders which I'll show you in our review app which is all built into the evaluation process that we do. And then once an agent is released or an AI tool is released uh you want to actually monitor quality in production and so that's also built into this whole framework.

So, I mentioned end users and making sure that they trust AI tools. And so, for this, I want to introduce the use case that we're going to be using for the demo, which is imagine that you are a doctor and you're about to see a patient and you've got like 30 seconds to figure out, maybe not 30 seconds, a minute, I don't know, to figure out what this patient who's recently been to the hospital had experienced in their hospital admission. So you know that they've recently been been admitted, but you don't have the details and you have to get up to speed really quickly. So I want you to choose between two agents. The first one you ask, okay, what was my patient's latest admission? And this is what it gives you. It says patients latest hospital admission details are as follows. It looks like they were admitted for being a newborn. It's a good reason to be admitted to the hospital. They've got private insurance. Uh their admission time shows this funky date. That's because it's deidentified, so ignore that. But um that's basically what you're seeing. The second one, you ask the same question. What was this patient's latest hospital admission? But instead, you get this process, this chain of thought reasoning where it shows you that what it's doing is it's calling a tool that I've given to this agent. Essentially, the AI is writing a query um or using a query that I've given it to get some information about my patient. then you see that the tool is responding with this um this result and then it's giving you the final answer. So it sounds maybe like I'm giving you a leading question like oh it's so important to have context but actually I'm kind of it's kind of a trick question. Like the point is that maybe if you're a doctor and you have those 30 seconds to a minute you don't actually want the whole reasoning. So my point is really that in some cases it's really useful to have the ability to see exactly what tools were called exactly what steps were taken by an a by an AI agent. In other cases it's not necessary. In other cases it feels like a burden. So the point is, you know, having this capability is important, but I don't actually know which is better in this case, which is exactly why we would iterate and test all of this with um clinicians in this particular case. Uh so with that, I'm going to switch to a demo. Uh this example that I was alluding to, I actually built with my colleague and uh what you're seeing here is a data bricks app. And this app is again you're in this scenario of your clinician, you're about to see a patient and you ask a question like what was this patient's latest admission? Um and you might get an answer like this. Um in this case we are showing end users the latest admission as well as the um sorry the uh the tool as well as the tool call as well as a response. I think I'm gonna have to put my mic down to type in my question. One moment. Um, but the the the next thing that that I want to show is that because this is an AI agent, um, we're actually taking multiple steps to get to the answer. And so the agent is reasoning about which tools it needs to call. So, I'm going to type. Okay. Um, so the cool thing about agents is that in some cases you actually need multiple steps to get to the answer. Um, and so in this case, uh, okay, well, we're getting an error because I ran this before. Let's get to the point. The point is that um, the agent can actually uh, use multiple tools in order to to give the response. Um so in the in the um slide deck here you can see kind of what's happening um which is that when you ask the question the agent is reasoning about what to do getting uh calling those functions that you see up there. So that first one I showed was getting patient admissions, right? And then the next one would be well actually the agent is reasoning, okay, I know what this patient was admitted for, but the question was actually more specific. I need to go and get call a tool that's going to get some clinical notes to give me some more information. Then I'm going to reason about okay, you know, is this actually enough to give a an answer to the clinician that's asking? Yes, it is. Okay, now I can now the agent can say this patient was admitted for these this and this complications. Um, so that final product that you saw on the application was actually fairly easy to build. So I want to show you how I did that.

So what do I mean when I say tools in this case? this case? So there's many different ways to provide an AI agent with additional context about our particular patients. Right? If I were to ask Chat GPT, hey, you know, what was Rebecca's latest admission to the hospital and what was it about? It's going to be like, who's Rebecca? What are you talking about? Right? That's general intelligence. When we talk about data intelligence, we're providing additional information about our particular data. There are a lot of different ways you can do that. Some of the more uh complicated ones essentially are to include or you can go into all sorts of rag architectures. You could vectorize your your various documents. Not that that's hard, but that's that's one way to do it. But it can be as simple as you've just pre-written a query that you think that the agent would benefit from and then you give it access to that data and that's all governed in Unity Catalog. So anyone who's using this is has permission to view that data. Um but what I'm showing here is a tool that I designed and it's actually as simple as that SQL query. It's just you know go get this information from this data set um where you know the the admission date is whatever the date is and whatever the um and whatever the particular patient is that I'm providing to the query and again the agent is reasoning about what exactly needs to go into that query which patient are we talking about. That's actually what it why it errored before was because it had reset. It had refreshed and so it didn't know what patient I was talking about. Uh anyway, so this is what I mean by a tool. And then what I do is I go into playground um which is our place to do some of that iterative development. And I pick a model that I want to be the foundation for this agent. And that could be an external model that could be an open source model like a llama model. In this case, I've chosen a GPT model uh and I've added those tools that I just showed you in Unity catalog. And then because of that, I can test this out in the playground. So that's that first step, right? Like if I want to build trustworthy AI, I want to eyeball it first. Does this look good? Does this make sense? Then I also want to actually view the trace. What happened under the hood? Do I think that what my agent is doing makes sense? And I can see here that it's actually calling that get latest admission tool because I said what was this patient's latest admission? So it reasoned, hey, I need to go and get the latest admission for this patient. And then this is what it output. So I can get some of that information about how the tool is actually being used. We also have built-in LLM judges that can tell me not only whether something is safe or relevant, but why it deemed it to be safe or relevant. Then, as I mentioned, um it's really easy to actually deploy this. So, once I've attached the tools, I can click get code and then create agent notebook. So, that's actually what I did to make this agent. I didn't really write that much code. I actually just clicked this and then it spits out a template that I can modify if I want to, but uh it's going to be get me really really close to the ability to just deploy this as an agent. Once I have that agent um I get something that looks like this which is a model serving endpoint in data bricks. So that agent becomes in and of itself a model that I can access. I can use it in the playground uh etc.

But in order to scale my evaluation and I think this is something that you've seen uh likely in in earlier sessions today just around the innovation with evaluation at scale. Not only can I see, you know, what one particular result had, but across all of the different times that I've tried to test out this model or across an entire data set, for example, a benchmark data set or a synthetic data set, whatever it might be, I can see how how often did I pass um based on LM judges, how often was I correct, and then in what case and why. So, I can click on this and start to really troubleshoot. Why was it that this was deemed to be safe or unsafe? Why was it that this was deemed to be correct or not correct? The other thing that I can do is once this is in production, as I mentioned, right, we need that long-term view. We now have the ability to see operational metrics like latency over time, but also performance according again to those judges. So, um correctness over time, what percent of my uh of my answers were deemed correct over time and how did that change? But of course, we also want to use human feedback and review app is one of my favorite parts of our platform because it's as simple as going into this model serving endpoint using using uh clicking this button and opening as a review app and what you get is an application with a essentially a URL that you can just give to anyone even somebody who doesn't use data bricks. So you would imagine giving this to clinicians to try it out to ask these questions and then I might go and give it feedback and I might say okay you know I I've I've asked this question I said what is this patient's latest admission why were they in the hospital what happened during their stay and then I would uh give it a particular response so either I could edit the the text itself or I can select uh you know what not only is it a good response but why is it a good response and all of that information actually propagates to the two inference tables which are themselves delta tables which you can then analyze. So the point is that this is an iterative process. Okay, let's see. Um so Um so I do want to take a second to try to actually show you um some of this in action now that we've seen how you would build it and how you would evaluate it. So,

I'm going to try to um ask it some questions and let's see what it says. Okay. So, that first step is I need to introduce it to the patient. Um so, I'm asking about a that newborn patient. Let's see. it's going to get that latest submission. Also, we're a small group, so if people have particular things you're interested in in terms of the evaluation process, I can also speak to that. So, feel free to raise your hand and jump in. We got one. Hit me.

That's a great question. It's actually the question I get probably most often when I'm demoing this. The truth is I did that because it's really easy to see in a demo. Um, and it's also useful in particular cases. So in some cases there are very very specific things that I know I want my agent to go do. In other in more more commonly what we see is people using like the Genie API. Um and actually Genie is now integrated into agent framework so that you can very easily as you're building an agent and um giving it context give it access to Genie which is then a text to SQL engine LLM to not only so essentially instead of it using a query that I wrote it's going to write its own query. Now why not just use Genie? Good question. um don't just use Genie for this case because we're actually trying to get information from clinical notes as well. And so when you have to interact with that unstructured data, it's actually like the clinical notes are in a column in my tabular data. But what this is able to do is when it reads that clinical note, it can interpret from that unstructured data what's actually happening to the patient. And so it's not just the result of some texttosql exercise. it's actually the result of um parsing out and and understanding and summarizing text. Um so if I were to ask Genie, I I actually it's funny because when I was doing this, I was also learning, right? So what I did was I actually went into a Genie room and I tried to make the exact same agent in Genie, but it didn't work because then I said something like, "What was this newborn's feeding plan when they left the hospital? and it wasn't able to understand that because it's not asking for uh that's not something that you can look up in uh based on the metadata, right? Based on the column headers of your tabular data, you have to actually read the content of the note to know what that is. So, you have to be able to to work with the unstructured data. Um which is one of the reasons why you would not want to just use Genie. The other thing is in a lot of cases you would actually vectorize your text, right? to your text, your documents. And so in this case, I'm not doing that because it's not really necessary. Um it's not large enough, etc. But the if you were to um have an agent in a real case like this, I think you would probably be using a vector database and looking looking up information that way efficiently. Yeah. Anything else? I can I can keep showing essentially like pieces of this and can't seem to type while I'm holding the mic. But um the but I could keep doing that. But yeah, any other questions? Yeah. Yeah, go for it. And if not also like I can do that for a few minutes. We can also move on to the panel. That's a great idea. Yeah, and it's exactly the point of being able to see in that evaluation tab that I was showing like where do we actually fail? Um, which is what I'm kind of gonna what I would show here as well is like when you go into that actual tab and the LM the LM judge or a human responds with some particular kind of feedback, you can then you can then start to understand, oh, it looks like actually my agent is calling the right tools, but then the tools are not really getting the right information. It needs different tools. or one of the things that my colleague and I actually built was um a custom metric that showed whether the agent was choosing the right tool and that then showed up in the evaluation tab and so it was an evaluation of was it even choosing the right tools in its process. Um and then you could figure out okay it seems like it's not choosing the right tools. Um, but yeah, if you had something specific like, oh, it's not getting enough data on this particular feature of patients, then you might want to build that in as a tool or you might want to use Genie, right, to get more generalized information. Cool. All right. Do we want to just go to the panel and let us all get closer to wine time? The wine in here. Cool. Let's do it. Yeah. You guys want to come up? Thank you, Rebecca. That was awesome. Yeah.

Okay. All right. Now, everybody has to stand up and move to the front. No, I'm kidding. Um, awesome. Well, we we know that it is happy hour time coming up uh here shortly. I'm Darcy Phillips. I am a director on our state and local government East Coast team. Um, and I'm joined by two of our awesome customers. Um, what I want to do is have you both introduce yourselves and give your feedback on that demo and relevancy to your customer base. So, um, Charles, you want to start? Yeah. So, I'm Charles Abraham, director of data analytics for Gainwell Technologies. We're a, uh, uh, state, we empower state and local healthcare to deploy uh, modernized platforms from claim processing to data and analytics uh, which is what we're, you know, partnering with data bricks and uh, have done exceptionally well recently. So I lead our product management group and working closely with our engineering teams to deploy uh what we call genius which uh you know is kind of fitting right it's a data intelligence platform so yeah perfect and Dr. Osbourne hey Dr. Osborne. Uh Tom Osborne. I am chief medical officer at Microsoft Federal Division. I'm also a full professor at Stanford and uh let's see what do we do? We do all kinds of stuff. As you know, Microsoft's a big place and well you were somewhere before. Oh yes. Okay. Yeah. So I was I was at the VA before and also doing a lot of stuff at VA that is relevant to this discussion. I started VA's first national center for innovation. Uh I had a big data analytics team as part of that. Was also director for our simulation center, deputy chief of staff, uh clinician, bunch of other things, but they all sort of revolved around the same goal and that's really about delivering the best most advanced health care possible and in this case to our veterans. Uh part of that we were using lots of different tools kind of tool agnostic but um you know before that I've been using AI for example for about 12 years but not really for the sake of AI or for the sake of technology just looking to try to really understand the problem very well and then looking for the best tool or tools available to solve that problem. Love it. Um well we we're talking a little bit about the session and you know risk and risk obviously gets discussed a ton. Um, but I wanted to kind of flip it on the head

a little bit and I wanted to start with you, Dr. Osborne. I'd love to sort of hear your perspective based on the work that you did at the VA. What is the risk of actually not using AI and innovation to improve citizen services and healthcare? I' I'd love to get that take. I Okay, great question. Um, I'm a balanced guy, right? So there's certainly risk and I think the the dialogue has been disproportionately about the risk of doing something new something very powerful in addition something that's moving very quickly and that's totally legitimate. I mean we've got to understand the risks because you know we in healthcare don't want to do harm with good intent obviously uh and so any powerful tool can be used for good or bad. Now there is as you suggested significant risks of not using powerful tools at least not using powerful tools appropriately. Now think about like what are the biggest challenges that we have in healthcare. There's no shortage of problems to solve in healthcare for sure. There's like I mean if you like solving problems it's a great place to be especially if you're mission driven. But if you think about it, like you could you could name them all off and you'd be here for days and days. But if you look at it like a high level, sort of like, you know, 2,000 ft looking down, it tends to cluster into different categories. And the way I see the categories, and this is debatable, but I see three categories. Three categories that are overlapping, and quite frankly, they are titanic forces that are colliding. Now I'm bringing this up for reference because AI is going to play into this potentially it could. Okay. So the problems that we saw uh we have a growing older population with multiple coidities and with that you have really complex care a lot of complicated care a lot of complicated data new research that's coming out. There's a lot to manage. So growing older population that's only growing. It's only getting bigger. It's only getting harder to manage. That's one circle in this three circle vin diagram. second circle. We spend a ton of money in healthare all over the world, but especially in the US, we spend $5 trillion a year in healthcare. That's 20% of our GDP. Like there's probably other things we can do with that money, especially with the outcomes that we're having. So, we spend a ton of money. Third is we don't have enough providers to take care of all those people in the first circle even if we could afford it in the second circle and all of those things are going in the wrong direction and where all those overlap is where we are and that is only getting worse. Okay, so you think about how do we solve that problem, right? And so the traditional way of solving problems in healthcare is just to throw more money at it which we don't have or more people at the problem which we don't have. That's the old paradigm. The old paradigm is not working and it's not going to solve the problem for today's issues. So you think, okay, let's go back to the first circle, right? You have a growing older population, all this data. The amount of data that's being produced in healthcare is growing exponentially. Now, you just think about peerreview journal articles. Like back in the 50s, a physician could expect that they would have to learn the same amount of data that they started to learn twice. Like there's a doubling time of about 50 years and that is taken an inflection point and now that replication rate, that turnover rate, that doubling time is 72 days. Wow. Wow. Like how nobody can keep up with it. So you need some way to manage all that new clinical data, the standards of care, right? That's just to sort of understand the baseline. But what about the patients? The patients are seeing more and more providers, more and more complicated care, more data, labs, pharmacy, different outpatient clinics. You've got people, specialists everywhere. They don't necessarily talk well together. We there's no way we can manage that with people. So we have these great tools if thoughtfully implemented can address those types of problems. So data uh clinical decision support that we just talked about we've heard about a type of clinical decision support really powerful it can be used for a lot of different things human in loop I think to balance that is really important but we have the ability for machines to look at a huge amount of data and clinicians that don't have time to look through volumes and volumes can have summarizations and insights that are pulled out and say hey doc did you think of this powerful Right. The other thing in that first category, I'm going to go on for a while. I'm sorry, but insights, right? We have the ability to extract new in clinical insights out of the existing data. We've done a great job at collecting data, all this data over, you know, decades, ever since meaningful use. But we haven't turned that data into information, that information into knowledge, and that knowledge into wisdom. So someone can actually do something about it. Now, we can. And not only that, with these tools, we can not only take that stair up into a virtuous location, but now we can actually have tools that can act on those things with us. So, we're taking it another frontier. So, innovation, being able to understand new things, new knowledge and wisdom. It is the most

exciting time in healthcare for new insights and innovations for understanding how people work in this complicated bag of chemicals and and proteins. So, okay, you talk about provider shortages, like there's burnout because of all the regulatory and drudgery that clinicians have to do, things that should be automated and it's just getting worse, right? physicians spend extra hours, you know, a day just on charting. We have tools, multiple tools that can solve that. If you don't do that, you're gonna have less physicians because they're going to burn out. They're going to leave. It's not good for anybody. Uh, another category that we talked about is the cost. That's expensive. Physicians shouldn't be doing these things. Uh, the overhead of just operationally like healthcare, healthcare is not very efficient. It's just the way it is. And there's a lot of redundancy. So the cost is going up, the overhead's going up. And when you have at a high level, if you have a health care system that's not using tools that result in better outcomes, better care, like who doesn't want better care? People have choice. If you don't have a health care system that can do those things, people will find another health care system. They will vote with their feet. And when that happens, market share decreases. those people who aren't using these advanced tools for better outcomes and better results. They're going to have less resources, less money, less market share, it's going to be a vicious cycle. This is happening in rural communities unfortunately and it's a downward spiral. You don't have the re you don't have the patience, you don't have the resources to invest in the technology, and you then have less patients. You don't have clinicians because they want to be part of the healthcare systems that's modernized. Mhm. So we are all at this critical digital divide that's we already have a digital divide but it's only widening right if we don't do something about it. Yeah. And we really have an opportunity to democratize that with the tools that we have in a thoughtful way. I love it. Well I speaking of risks you know I do think that gain well you know you guys kind of took a bet you know four years ago. Obviously you have a tremendous you know wealth of expertise and knowledge built over 50 years but um Charles I'd

love to sort of hear how has the decision of adopting a nextgen platform like data bricks four years ago um how has that changed the way that you support your customers now has that changed the way that you think about how you can provide trusted solutions for them so that they can you know do the things like scale into this next phase of healthcare. Yeah. No that's And to piggy back off of uh what Dr. uh Osborne was saying was, you know, when I work with states and uh as they look at AI as well, um just to wrap that up real quick is there's a fear, right, of AI and rightfully so. Technology person, I tell my kids, I was like, don't trust AI. Not yet. Not yet. Right. And but I'm like when I talk to states, it's kind of the same thing. they're like a ease or you know kind of a worry but there are some that are like hey I'm going all in right and to those I say great they should because you know especially now with the government you know from a government point of view with efficiency and tightening budgets it's huge right yeah um and you know honestly that's where Gainwell has really helped empower the states with what we've done with data bricks is uh you know like you said we took a bet uh bet or you know we took a strategic choice about four five years ago as we became gain Well, yeah. Um, and you know, 50 years of history in state and local healthcare, it's just like the VA, right? Is a lot of antiquated systems. Um, but you know, our company has really focused in on trying to turn that, you know, I was going to say the Titanic, but that sip shanks. So, we'll say it's like a Royal Caribbean icon of the seas, right? You know, shifting that way. And it's kind of exciting when you get in front of a state, you know, Medicaid director or uh CTO and you're just like, "Hey, I'm on data bricks." They're like, "Well, first, you know, you're the monolithic company." Yeah. But as soon as you say data bricks, they're like, "Hey, okay, these guys are understanding where we're going, what our vision is, and how we can, you know, really help them with health equity, population health." Um, uh, big thing is, uh, predicting fraud, right? And as we've developed models and things to empower the states, utilizing that 50 years of experience has been huge. And being able to execute fast is the other main thing that helps states understand, hey, you know, gain well can help them come in at a better price point because of the costs that are related to it. So yeah. Well, I want to, you know, I know we're we're winding down, but I I want to kind of probe on the fear aspect because I'm I'm really curious for your perspective and the audiences. You know, I think when I talk to customers and they speak about fear,

it's it can be kind of a wide array. It's not just fear of the technology, but fear of where to start. Um, and I I'd love to hear, you know, both from your perspective, Charles and Dr. Osborne, like how how does having kind of a a foundation, the right foundation for your data in place, like how do you kind of navigate that talk track with your customers to help them understand that they can innovate in a safe way and and not I I think it for me that's what I feel like the fear is often we're overwhelmed. We have so many things, you know, going on. Everything's still on prem. Like the idea that we could, you know, leverage what Rebecca built is like a pipe dream. How do you help them overcome that fear? Um, or do you think that is not as big a fear and it's more the technology that stops them from innovating? Yeah. No, I think it's a little bit of both for states, right? We've got states where they're on one end of the extreme where they're like, "Hey, I've got a bunch of bad data." And, you know, in state Medicaid data, it's like, you know, there's being a data person, it's garbage in, garbage out. But, you know, from a claims processing system, you know, and as states have shifted to a focused managed care, uh, managed care organizations have zero incentive, if you will, to send quality data, right? So, you know, sitting down with states and they're like, "Hey, I've got a bunch of bad data. You know, what can you do?" And we're like, "Well, we can do a lot of things, but what's causing it?" Right? You know, and so we're like, "Happy to show you all these solutions, but, you know, let's first take it at the data level, right? because from AI to understand the fear, you've got to start with the data and that's where you know working with them through that process has really been exponential and seeing the return on investment for them. Yeah. No, the crawl walk run I think that's what's so important is I think a lot of customers just want to run it and and you know vendors are happy to meet them. They're happy to go all in on some widget and everybody spends a ton of time and effort but it can't scale because you haven't taken the effort to do the leg work first. Dr. Dr. Osborne, I'm curious. What was that like for you? Um, you know, when you were at the VA, was fear something that got in the way of what you guys tried to do a good bit? It's a good question. Uh, I think it depends on the stakeholders. Yeah. You know, everybody's different. You know, a lot of people I think a lot of people just want to solve problems and they care less about the technology. They just care about like how do I help my patient? How do I get through the day? How do I see my kids soccer game tonight? Yeah. you know, how do I not have a 12-h hour day every day for the rest of my career? So, those I mean, that's those are other fears. Uh, but certainly no one wants to do something that's going to cause harm. Yeah. And the unknown. I mean, you talk about like the greatest fear of humanity, you know, you go you go see some movie, you know, and it's oh, the scariest part is the shadows, you know, the thing in the behind the closet door or under the bed, but then it comes out and you're like, oh, it's that stupid mask. And so it not knowing. So I think a big part of this uh journey of decreasing fear is bringing people along on the journey with you to understand the technology and it's tough to do because it's moving really fast. Yeah. But to understand and and AI is not just one thing. It's like a thousand different things, right? And to to say, hey, let's start with the problem that you care the most about. Let's try to understand that problem as best we can and then look for the best tools available to solve that. and let's really all understand how this tool works and how it can work in the context of what we're trying to solve for you and how it can go right and how it can go wrong and how we're going to mitigate that. And it's a partnership. I think when people have something that's thrown at them, they have no control over. Sure. They don't have a chance to to weigh in and it's really tough to buy in when you don't weigh in. And you mean

partnership between IT, security, business, kind of getting all the all the stakeholders together. Yeah. Okay. That's a that's a great topic. Can I open a can of worms? Oh no. Okay. So tomorrow Yeah. Tomorrow's session we'll be discussing that. Yeah. Different stakeholders are going to have different fears. Yes. How is that? And they usually don't you know in any organization I'm not just talking about government but like any organization you're going to have let's say for example you know you have your chief of staff, you have your chief of department and they're all going to be thinking about different things. Then you have the individual clinician using the tool. Then you have it and biomemed and security and compliance and legal and like they're all coming at the thing from a different way of saying risk benefit like it's better to do not right do nothing is the biggest Yeah. You I built a committee where they all had to sit at the table together monthly. It's great for job security to do nothing. That's true. That's true. Until the until the business goes out of business, you know, and you lose all the market share and then That's right. It's not good either. Yeah. Well, I I joke often that our biggest competition is do nothing. Um, so yeah, I it's nice to hear you both say that. Well, I know we're coming up on happy hour, so I think I'll I'll wrap it up with one last question each. Um, you know, as you're

looking to the future um the future of Medicaid um and decision support systems that you guys provide and other areas like what are you most excited about and what are your customers most excited about? Like what are they what's getting them excited right now about AI, ML, and Gen AI? Yeah, I think for for us at least it's uh I say some agencies are they're like a decade behind like and so coming to some other I was trying to be gentle but you know I think you know coming and working with other industries and seeing how fast it's moving you know it's I would say they're excited about what everyone else is finally excited about whereas it kind of moved in kind of slow motion for a long time I I see them now saying hey I can chat with my data but how do I do it right so I see a lot of the you know even automating agentic and things like that where things that they were doing now they don't need as much staff right so it helps reduce their budget reduce what they're spending and they can focus that on things that matter right is how do they solve those problems and improve yeah exactly yeah awesome Dr. Dr. Osborne, what about you? Yeah, like I said, uh, good question and it'll depend on the stakeholder, but if you just think about a like a clinician, for example, there's sort of a threephase level of excitement and it sort of ramps up a little bit. And so, at some level, you know, you have a physician that just wants to get through the day. Yeah. And they are being bombarded by bureaucracy that they weren't trained for and they don't like. And so, they want to work at the top of their game and that's not charting. Sure. And if they're able to spend more time with the patient, then that's going to result in better patient care. So it's good for them and good for the patient. And then if they happen to have more bandwidth, then they can explore the questions and work to solve the problems that are intractable that really bug them like hey this one disease or this one condition or why the patients whatever and so they have the bandwidth to actually solve those problems and the tools that we have can help at every step of that journey. Yeah, I love it. Well, I'll open it up in case there are any questions um to our wonderful panelists. If not, we will not uh keep you from your wonderful happy hour and uh dinner plans. But thank you both. Um and Rebecca, thank you. That was an awesome demo. Anyone have any questions for these guys? No. Yeah. Padu, I see you looking back there. [Laughter] [Laughter] He'll send you an email. Oh, yeah. Um yeah. Well, thank you guys so much. I really appreciate you joining us. This was really good conversation. Thank you both for the partnership. Um and enjoy your evening. Bye. Thank you.