Your Data: Fair or Foul? Public Sector's Secret Weapon

good afternoon everybody for this next session it is my pleasure to introduce you to Milos and pritesh who will tell us how the lake house platform is helping customers in the public sector so or to you guys thank you thank you Sergio um so yeah welcome everyone [Music] our talks on Fair or foul data public sector lake house as a fair platform we'll go through the principles of what that is I'm here with my colleague milosh I lead our UK public sector team um to get a partner on driving both the sales and the technical vision for our public sector in UK and I excellent so look what we're going to cover today is you know what is fair it's around that sort of findability accessibility we're looking at interoperability and reusability and how that links to the lake house platform we think it goes very neatly hand in hand and we'll walk through that for the rest of the presentation

so what we see Fair it is a cycle uh is a cycle around that um making sure when you're building your uh data life cycle that your your data is findable um that your users can find it and access it appropriately the right permissions are granted and then through that data life cycle as well it will need to interoperate with other systems and then finally as that data product product or data lifecycle matures it will need to be used by other people in a public sector context it's quite uh obvious from a policy perspective policy implementation perspective that this is done um and and it also we've combined it in this sort of introduction as well for building good data products in general so the Pyramid of of that really starts with the usefulness of the data that you're producing um is it explorable and searchable as well um the Simplicity the scalability and I think almost when that maturity happens The Habit generation is what we want to achieve right so you get those foundations correct that both your internal user external users almost habitually and seamlessly you can access that data and use it for its intended purpose um but we'll talk through more how we actually look to do that I think you need one more sled sorry yeah sorry yeah so yeah and um

from the product perspective as well when we are operating in in government context or in general in a regulated Industries and with a lot of restrictions and constraints and policies that enforce the behavior how do we map then the the product design best practices into that system usually what we see is actually the best practices are very transferable across Industries in a lot of policies and advice has been built thinking about those best practices they just may not use always the same terminology they may also not always put it in the same shape and form so here what we're trying also to illustrate is that parallelism you see this cyclical notion of the product being introduced to serve a purpose that's the usefulness part and then it matures and grows as long as it promotes best practices that's the The Habit generation bit and at some point it eventually it declines because either there is a better product or the the problem has been sold on the um right hand side you see the the government proposed flow for building the good data products and while it looks very different it actually follows very similar implicit cyclical nature so you need to plan and you know identify the need and the purpose of the of the products once you then collect your data you build those data products and then you use them and process and the process actually generates new products and new usefulness so there is an implicit feedback loop there it is just not Illustrated on it and can we go to the next slide please what is also another way of looking at data products in within the domain is through a lens of economy

and I think we're with the sustainability taking a quite a central Topic in in many Industries and many discussions we talk about this linear principle of economy as a legacy way of observing an economy where we extract the resources we produce the products and then eventually once we have no use for them we dispose and the similar thing actually happens in Legacy Platforms in the data so that's why we call it a linear data economy where we generate data we refine we filter we we use it to make decisions the differences with the this virtual product is that we actually accumulate a lot of it you can't just delete the data after you have made the decision about policy or um um or legislations you need to keep it for audit purposes and for um historical View next one next one yeah and and this is the important thing right that can be quite easy especially in a government context when it takes a fair amount of investment to get some of these things going if you look at it in a flywheel concept for example building up big infrastructure projects any sort of large data project can can feel like a while to get going and then it's quite easier to fall into the linear aspects of things as we showed in the last slide but it's important to keep this circular motion feedback loop going and applying that both from a product perspective in general or program perspective as well and then into your data products built within your your environment and settings so that's really one of the key takeaways here that we wanted to express

so starting with the findability in Fair so so one of the things that people have to navigate is their own organizational structures so the way that you're set up be it centralized or decentralized has a factor in how you can go and find that data and in both modes there are opportunities and approaches to do that but typically we see the barriers right so if I look at certain centralized organizations they can hold a real moat or guard around that data often for the right reasons for security for governance but everyone has to come to the center and try and get that and then sometimes it kind of that that organization is a is is kind of evolving all the time and this decentralization can happen as well where people go out and build their own silos and their own kind of versions of those organizations and extend that to a data data perspective as well because they're not getting what they need from a centralized perspective sometimes that's actually done intentionally Federated as well so the structures if you look from a federal to localized government perspective is one one such scenario the other ways around these kind of like you know the Greenfield and and the Brownfield right so milosh touched on the Legacy element of things and these things have been around for since the beginning of time and we're seeing a lot of people in government going well okay I'm not going to be able to sort of do much with the stuff in in the Brownfield let's go create a Greenfield and start from scratch and it's quite easy to build new technologies and and start fresh however some of the critical data infrastructure and key assets for government exists without Brownfield so it's about Crossing that divide and finding ways to interoperate and what we're finding is that kind of stolen a bit of Thunder from interoperability but finding ways in which to make that happen is going to be crucial and key and we believe sort of Open Standards under underpins that but we'll go into that more in a little while and then finally sort of this analogy here I once had a government data scientist tell me uh his frustrations of having to come in Via what he called the front door to get data from within his own organization to do the work that he needed to do for uh policy Improvement and policy decisions and you know the work he was doing you know was contributing towards some really important Net Zero goals and for the department and you're kind of going why is he a sort of almost second-class citizen because of the Federated and decentralization of different departments within that one main Department um so yeah we've heard this sort of uh analogy before and a few few situations so what's the solution my life yeah yeah what we found in practice to be the

solution for a lot of this fragmentation and um discoverability or findability as you put it in fairies the the centralized governance and through Unity catalog and through a simplified way of handling that that governance piece of who can access what so you can still keep your data decentralized across business units or teams directorates Etc but then keep that access patterns unified through a standardized way of operating on permissions and why is that very powerful is that you have concepts of internal tables tables that are fully managed by by unity catalog but you also can manage external tables they have been built by other tools and finally you can with some recent announcement you can also manage files directly on the on the containers within your cloud storage and this is some things that we've walked through in private previews and while the the evaluation has been going it's proving quite interesting for many specialized use cases where you have to operate on custom formats so we don't enforce you into a place where you actually have to convert things which also breaks apart the the principles of fares you it makes things much more complicated I think the cherry on the top is the how easy is to actually manage the the permissions It's a grand statement so it's very declarative it's based on SQL standards it's it's much more accessible from the perspective of you know the people that need to manage those permissions can do that very easily if you top that with the fact that we have also terraform providers and other open Technologies it becomes much more scalable so you can also build scripts around it and onboard whole units in a very easy manner ah and that allows us also now that we have that single management layer that combines the permissions and and the listing of the data you can go and search through all of your estate and if you have permissions to access that data they will appear in the searches and you can easily Discover it and go from there

so logically next is accessibility but actually we just want to sort of muddle up the the fair acronym to start with interoperability um essentially the interoperability mode when you're trying to work that out in your in your systems of data and you look faced with something like this which is you know a heterogeneous number of systems um with different uh formats different uh colors shapes and sizes it can seem very difficult to work out how to bring one system to another let alone multiple systems talking to each other in turning your organization so where would you go you'd normally start to sort of categorize this into um sort of these these sub verticals or sub categories sorry where you go right these sort of databases can talk to each other so we bring them together organize in a very structured way but then essentially that lets certain types of unstructured or semi-structured data left out in the cold um and also like you know rigid kind of Frameworks of connection which don't necessarily work in practice as well so what you need is I mean we're trying to think of an analogy for this an organization of these different Data Systems with mapped out interconnectivity so I don't know those who've been having a lot of coffee this week but this is the uh the the chemical module for caffeine so uh you know we thought it was a good neat kind of arrangement of those different molecules and then the interconnectedness between them and that's what we need to sort of strive for is that organization but interconnectedness and uh read the lake house platform provides a really good opportunity to be a home for all different shapes and sizes of data within your organization

be it unstructured semi-structured streaming you may have seen this sort of lake house platform before so I won't go into much detail but I think underpinned by the governance that you have whether you need to catalog and also a ubiquitous sort of format within Delta Lake I think is you know and again some of the announcements are on Delta Lake 3.0 further reinforces you know that ability to interoperate furthermore so that's what's really exciting there I think again reflecting on some of the keynote this morning the contributions to the lake house platform and its components from an open source perspective are really significant that was talked about to a great deal of extent but we see common formats as well furthermore within government really focusing on things like Sequel and R as well and we want to look to make those you know first class citizens within the Lakehouse platform as part of the ecosystem and finally sort of some of these open source projects that we've been able to contribute um for an emea public sector point of view so check out our projects around Arc which we've used around entity resolution and doing that in an open source manner we worked with Ministry of Justice to build that accelerator and take that to a variety of different government departments we've used kakapo with a kind of financial services regulator kind of lens on things and looking for that anomaly detection that you need to do in the structure of the market I don't even feel a comment on Mosaic it's very topical yeah sure and Mosaic we used in a little bit more horizontal perspective not to be confused with Mosaic ml this is a geospatial package and because geospatial is quite pervasive in government you you have it almost in in any Department we we actually built it with from ground up with one of our major customers ordering survey so we started as a workshop and then we built a a whole project that's now been used by more than 150 organizations worldwide so it's quite an and this is from a local team and I think that's also very important that while we as a company have that ethos of contributing to the open source we it actually permeates into local teams that are aligned to different Industries and we really strive to work together with the customers and and and and and if something doesn't exist on the market build it together and another one also very interesting would be dolly um I think we're having now this acceleration around these foundational models language models um large language models and I think where we're seeing the shift is uh these will be accessible very soon to almost everybody the differentiator will be your own internal data things you want to keep the control of and that bring technology to your data into the lake house this is what Dolly actually enables you to do you can keep your data which is your most valuable resource your IP and then bring the modeling techniques to that data and learn from your local data and find unit and everyone just wanted to make points

around the cloud I think one other very important thing around interoperability is not enforcing a single Cloud not enforcing any vendor lock-in so we do provide our our platform in exactly the same way across all of the three major clouds so that's very important because we will not force neither you nor your partners into a specific single architecture on a single vendor and to to you know put the everything into into full perspective obviously and this Alliance very closely to that molecule of coffee all of those different pieces need to work and interoperate together for that you actually have a wider ecosystem there are solutions that have been part of your platform for years or decades and we actually interoperate with many of those um technology is now this slide is actually too small we had to make quite a significant cut we're talking hundreds and hundreds of other partners it's just a you know a sample few for specific different types like data ingestion like machine learning and also your Regional or Global sis that help you deliver those use cases to your teams it's really important to you know work together to deliver that vision of a data platform for a for public sector and for the um um uh good of the citizens 100 I mean I was really enthused going around the Expo a couple of times this week and and speaking to all the different uh Partners we have out there um really out of the box it's quite easy to integrate with data breaks but you know they're very keen to invest um ways in which to walk to the lake house platform knowing how open we are making it easier for their customers to use their products that are specialized in some of these areas you know I think databricks doesn't necessarily take that approach of having to be all things for everyone so a really rich ecosystem coming together I've seen it a recent AWS and Microsoft events and and now this week and it's been really great to see that that ecosystem coming together to build those outcomes for our customers and you know a host of Partners as middle shed as well in terms of implementing that from from the global ones that you see up there and even local ones that we work with as well in addition like you know your kanoses and your advancing analytics as well

yeah we're coming we came back to the accessibility accessibility um so I think one of the important things about accessibility is also how you have built something how is that platform in general evolved um a lot of Enterprise events if we step uh you know abstract a little bit from the public sector Enterprise as well they have started building Platforms in many different locations to serve certain use cases and as the time evolved they had to bring everything into an umbrella I think in government that's even more um emphasize because of the specialization of different departments to deliver a specific purpose and then you may have also directories under those departments again further focusing uh on on a specific delivery an outcome there may happen to be a very organic way of growing a platform and then trying to make those things now form a a unit at the end and that can be really a daunting task now on the opposite Spectrum if you start building all of your capabilities from the best practices in mind and that's one of the reasons why we're talking about good product design which is not necessarily something that usually you hear when you talk about Technologies you start with architecture diagrams rather than a product principles but it's really important to start from the best practices from a lean design from following the standards that others in your ecosystem are going to follow because then you have that um planned growth you know all of a sudden everything kind of aligns and and and creates that um Symmetry and and coherence yeah like our t-shirts exactly

and then um expanding a little bit on that centralized versus decentralized view it's it's very important when you have that organization is how you share the data then so it goes back to interoperability when we're talking about accessibility and one of the common themes in this whole consideration is no ladder is decoupled fully from the other so you when you think about one concept the other one kind of creeps in but the way we share data obviously is governed by many things sometimes about the classification of the data and you you shouldn't share and we are investing heavily in addressing some of those uh Concepts through concerns through Concepts like clean rooms and in other cases it's an open data and you actually need to share it even outside of the the public sector domain and this is where we talk about this distributed data sharing Network where you may have a combination of technology Partners regulated bodies Regulators all needing to share data in a different capacity between each other and they all may reside in different Cloud deployments some may even be still on premise it becomes a very um challenging and messy uh situation so if

we go to the how we how we go about solving that so our Focus was in designing open Technologies we've said that several times but to fortify that we we came up with this protocol called Delta sharing so key components of it are all open source so Deltas as a former Delta lake is an open source project it's one of the the most popular open source project then we use that format and build an open protocol on top of it that allows you to easily share data from your Lake to other organizations while maintaining a full control everything is uh is built from the security principles first so we have a lot of collateral out there blogs and and documentation covering all of the things you can do from the security perspective so you know that you're not over sharing and you're not leaking anything but what that allows you then to do given that it's an open protocol and it's not it's um agnostic with respect to which cloud and which platform you can share between organizations within an organization and you can also share uh often on cloud meaning that these Services because they're open source can be installed on premise and you can use them to share data into cloud and uh accelerate some some migration use cases for instance the best part at least in my my opinion is the fact how quickly that got adopted by the community so we have to put these three dots in the slide where we can't list all of the connectors because they are being new ones are being built every day and it's already quite a big list but we have support for ingesting data as if it was your local data so you don't really need to think about this on another server in Python in Java rust C plus plus node.js and many other languages are coming and also in your uh very um um common tools for bi like power bi Tableau one of my favorites is also Excel I talked yesterday about that but that's one of the certainties of Life Excel will never go away and yeah next slide we've used then that set of tooling to build the marketplace and you've heard an announcement this week around that going uh GA so the Delta sharing is the building component for the marketplace and what we talk about here is not just data it's any product use built using data so we're talking about derivatives as well models dashboards notebooks code all of that can be shared using the same mechanism um and it's you know there is more and more mandates coming down we'll touch on some of these uh from from top down in government to share that data across each other at the same time you know they come across the challenges internally about what are the rules of sharing from one one organization to other we believe that this can help solve both the Delta sharing and Marketplace Concepts can help overcome some of these challenges where you can fortify and justify who owns the data where that data goes as part of the lineage and provide you know the regulators and the compliance people on both sides the comfort that they need that you're looking after the data and the way it was originally intended as well and one more um concept within the exchange of data would be that of clean rooms I think very Central to what we do in the government because in many cases you'll have data that's classified at um um top secret or restricted type of classification how do you then extract Insight from those assets across departments without actually oversharing or exposing data in a wrong way clean rooms you can think of them in a high level as a one-off disposable compute you can Delta share data into it but no individual party can interactively access the data data will be executed against the code which is treated as a logical contract these are the only operations that will be operated on it and if the results are compliant they can either be shared to both parties or only to the one party you can enforce a different classification to the results than you did to the input data again with respect in the policy in place quite a powerful concept of also for organizations that have a hybrid design where some of their data had to remain on premise even while they want to use Technologies available on the cloud because the data will only ever exist on a fully encrypted Channel and then be completely disposed after

and really we're on to the final uh letter of reusability and and also the slot we put in is reproducibility as well um um it's about really moving away from you know you can have some really wizzy Ferrari or Mercedes in this case with uh not being sure what's under the hood but actually making sure that we build these data Platforms in a way which you can see the internal componentry uh and even after the fact it can be audited and looked looked back on for that transparency purpose is is super important within a government context um and this is where that lineage starts to sort of really play a role making sure that you see the line of sight and where that data goes through its Journey yeah exactly and it might not be an obvious reason why we're talking about lineage in a context of reusability and reproducibility but I think it in a perspective of in order to use or reuse or reproduce something you need to know how it's been built so it's very important when you are consuming data that you trust that data and again how can you trust the data if you don't know how it has been constructed so lineage provides you a very granular way of viewing how did they come up with this value in the in the end the value that I'm sharing with somebody externally well through going through these steps and one of the powers of the platform is that actually anything you do in databricks will be tracked and you will have that lineage came from this table combined with this table this filter was applied and as we further evolve the platform Concepts like we have applied this machine learning model in this step and you can inspect which model will come into play and yeah I said the word model so

obviously the next slide is about models we have ml flow as one of our other very successful open source project this is another protocol focused on the governance around now not data but Concepts that generate data so machine learning models will generate data they will generate predictions they will generate forecasts so you can think of them as something that produces data based on observed data again in order to use this to make material decisions you again need to trust them and in order to trust them you need to have a very good governance and audit Trail and explainability of them so we built ml flow in order to track how a model has been produced and it's it's not a simple concept as we talk about models in a almost like a aloof way but you actually need to know which data at which point in time uh which parameters have been used how many of the experiments have been run I say the best model how do I know this is the best model so how I evaluated that all of that needs to be bundled together in order to actually go and make a material decision that can actually you know influence a policy or I don't know I used to work in finance influence if somebody gets a mortgage so it needs to be really tight ship around that so ml4 provides you actually a very easy set of apis to get all of that and inside of databricks any model you train will automatically get all of these things tracked and linked and you can just click through and see all of the concepts that I've listed and really that brings us back to you know know we've gone through the whole fair fair concept we've brought some of the parallels in terms of ways we think we can help solve and make it life easier to apply that fair context into building your data products into keeping it a circular motion not a linear motion that life cycle as well that you talked about right um yeah and I would say for me I know it's a busy slide there is a lot of Concepts and I wouldn't necessarily say any of them is more important than the other but I would probably single out the Habit generation I think that's really important that anything you build is easy to follow and there is a pool to follow because you know there is a value of those decisions and those actions so always focus when you're designing platforms and solutions that into that idea of it should promote the best practices and less standards yeah and the value middle she talked about taking that thread forward here right so the value's been driven from both bottom-up we can see the community coming together uh our partners from a technology perspective as well but but also from the top down government are realizing the importance of getting this right um you know we had uh the national based strategy from the UK published a couple of years ago and that's really being doubled down on ahead of focus and the rise of AI but making sure that's done in a in a safe manner for the for the citizen to give reassurance so here's Rishi soon I've met um Joe Biden recently and in the Autumn looking to do an AI for safety conference where they looked provides on those principles and I can better bomb Bottom Dollar that fair will kind of play a role into that concept and we've seen this from a federal strategy data strategy

perspective as well embedding these principles and and kind of patterns of interoperability reusability and making sure it's accessible to the right people the right levels and finally as well with our Australian counterparts in government as well similar Frameworks exist to reinforce this so the value is uh definitely seen and understood top down from government it's now about sort of getting that out there and and rolling it out and really why we need to do it from another perspective is these insights and policy decisions lead to Citizen outcomes in the last three years we've seen a number of activities have happened in a very short period of time from the likes of covid through to the inflationary pressures through to some of the conflicts that have happened that have meant that governments and and citizens together have had to adapt to quite extreme scenarios very quickly and not having the right sort of principles set within your data leads to sometimes less transparent decision making incorrect decision making because they don't have access to the data and you can see here like less than three percent of that data has been really used to make that critical decision making we really want to see that um um grow in in kind of value and importance as people start to adopt more better best practice within their data both in the Brownfield and new Greenfield kind of scenarios and and I think that transformation of that Legacy and also going back into the Legacy and taking that data is is critical to that as well as driving the modernization programs that you know if you're here in the room you're probably part and parcel of driving that modernization program within uh within your Department's day today and also like the ownership of that data Mirage touch on this earlier as the rise of gen Ai and llms grows it becomes even more important that you're able to give that transparency and that Focus um it's very small up there but like you know Italy was one of the first governments to actually ban the use of chat GPT um just because they you know don't feel comfortable about where that data sits and lives and what its usage might be so I think this is you know we're going to see some of these reactions from government entities when they don't feel ready whereas if we can work within a very um ubiquitous way glass box approach explainable data models and data products that we build governments will feel confident going in front of citizens and putting that in front of them as well to say these are the solutions and the ways out um and and that depth of knowledge is going to be critical from a skills perspective as well to sort of generate those models um in a way which aren't hidden to you know your everyday citizen and finally look I'll leave you with you know a few points here what is the opportunity from um from this study here um whether he talked about the opportunity to engage citizens more um to go and get some of the best talent out there as well because the missions that we drive through government are unique across any other industry and the impact we can make from a Citizens outcome perspective quality of life perspective the policy initiatives we make both for domestic and international policy can be really critical for the outcomes of citizens and feeding data into that in both the transparent way in an effective way is going to be crucial and important um and and yeah generally grow the efficiency like uh like it's happening across a lot of different Industries within government is the opportunity that lays in front of us so you know we're going to need to do it together as part of a whole data ecosystem and we look forward to playing our part and working with a lot of people so with that say thank you and uh happy to take any questions foreign [Music]