Unlock Your Data: Databricks as Your Enterprise Serving Layer!

My name is Ivan and to keep it simple together with the other Ivan today we'll uh share our journey at the bank of how we decided to build that infrastructure to meet the needs that we had. As a reminder, there will be uh a session uh survey. Please complete that after the session. Thank you for that. So at the World Bank, our mission is to

uh end poverty and boost shared prosperity. We do projects that we we do hundreds of projects in hundreds of countries uh every single year. And to do that, we really need data. So datadriven decision- making is at the core of almost everything that we do. For the past 60 years at the World Bank, we've collected vast amount of data being that structured, unstructured, semiructured. There are over uh I always keep on forgetting these these numbers. Uh there are over 10,000 applications that current work and need to use this uh this data in all kinds of forms and fashions. Again, we have over 300 terabytes of data that we process uh daily. Our overall data storage is probably somewhere around uh like the upwards of 30 pabytes. And this is not everything. This is just the the data that I have kind of been able to to see and work with. When we started our cloud journey several years ago, we said, okay, so there's that much data. We need to figure out how this data is accessible. how do we eliminate these data silos and with but when everyone started kind of moving to the cloud these silos just kept on growing and growing so I said okay what are we missing

couple of years ago we decided okay so let's do our forward-looking strategy and focus on six things let's remove these data silos but at the same time keep an open and uh democratized framework Let the applications and application teams and whomever needs to work with data work in the platform and the system that they want to do that in. but also enable them to be uh to bring their data in a centralized platform where you have out ofthe-box capabilities such as data protection, such as integration, such as making this data out of the box available to everyone in that organization. Uh we couldn't really forget about scale and performance. So that was at core and

when we started looking at what was available on the market we kind of naturally uh decided uh to choose data bricks not only because it was really easy to get on board with a platform but because back at the days it was secure it was integrated. It is quite agile and the time to market with whatever you start doing on data bricks is extremely fast compared to everything else that we've been doing. So we said okay we place data bricks at the center and if we need to bring some data in we bring it in we expose it through catalog but how do we make this data available to everyone else? what is it? What is this other thing that we need to make also available in that platform so that it's the integration is seamless and everything is protected before we get there. So we identified

generally three uh large use cases of what we needed to solve. So if you're uh on uh on our serving layer, your downstream applications could have been any data products that you might be running, any data cataloges that you might be running locally or any visualization and enterprise reporting. So by the way fun from fact uh if you Google today corporate scorecard data 360 or finances one these are all platforms that are available as public good you can download data out of them and it's all powered by database the second very important scenario was self-s service I mean everyone wants self-s service everyone wanted want to do their reporting or data preparation themselves they don't want to depend on a on a central team so this is a second uh big scenario that we needed to support and of course uh well in our case uh just to mention but Ivana will be sharing more uh in a bit we're talking about powerbi we're talking about artics we're talking about tableau in these scenarios and of course we have transactional applications some of them needed to consume data out of data bricks how do we make this available datab bricks is a well it's a big data processing engine so can we really scale it to meet application demands and of course everything needs to be protected government governed and made uh secure just available out of the box. Most of the times if you work on the platform you have to do all these additional governance and privacy things additionally and it's often uh overlooked. So we decided okay let's bring it as part of the offering and let's uh let's just allow people to use it out of the box slightly more on the technical

requirements. So okay so because you know these three large scenarios were quite I would say uh complex and complicated but we decided to follow best practices and uh chose the most critical the uh most sensitive and most protected use case to get started with. Uh so we had to focus on three major I would say deliverables. So these scenarios in the AIB world this pretty much meant we needed to deal with hundreds of users simultaneously getting reports in PowerBI using the red query to data bricks over a fact table which had more than a billion records. At the same time render time needs to be less than 10 15 seconds sometimes even less. When it comes to applications, we wanted, you know, so we had applications and all of them pretty much consume data over JDBC orc or REST APIs. So we had to make sure that this is available, supported and performant. We wanted to allow uh external engines to be able to bring their data in oracles, SQL servers, whatnot. And last but not least, enterprise integration. The one thing that is again often overlooked when people choose platforms and solutions entry integration was a key because active directory permission management unless you streamline it then uh you set up yourself for failure. prolevel security, object level security, masking, everything needed to be out of the box and of course integration with other enterprise platforms uh running on the back end such as uh Splunk had to be available out of the box. How did we achieve all this? So I'll pass the button to Ivan to share more on the lessons that we learned and how we solved some of the challenges that we had. Thanks Ian. I hope everyone can hear me. Okay. So, we covered our technical requirements and our forward looking strategy. So, now let me tell

you how we did this in action. So here on the screen you can see uh the building platform the building blocks of our platform. At the bottom we have our open data lake where we are storing all the row data and all various data sources. Then on top of it we have uh the delta lake uniform layer. We make sure that all the data that sits in the lakehouse is uh using delta lake format because this ensures we have ACIT transactions, we have time travel capabilities and also we can enable the uniform format which can then be accessed by other engines like hoodie or iceberg. Then at the center of our solution we have unique catalog which is our centralized governance for the lakehouse and of course it helps us with the discovery with the access control lineage and of course auditing and monitoring. The next layer is I would say these are the three different serving tiers. The first one is the quasi compute. This is the tier which is sitting inside our cloud premises or uh by our our cloud premises I mean our Asure tenant. Under this category we have the the interactive cluster and the the regular uh SQL warehouse. The next one is the serverless compute tier which is pretty much all the compute which is uh provided by database. Here we have the serverless SQL warehouse. Here we have the serverless compute for notebooks uh the model serving etc. And then last but most least we have the API services tier which we are using for either delta share or for our custom integration with Asia API management. Of course on the top the most important layer I would say this is the network and workspace isolation layer we shouldn't compromise on compliance so we make sure that everything is well isolated and we are following all best security practices for example all our workspaces are isolated we have IP access controls we have agress controls we have uh private endpoint points and all the goodies that data bricks provide us. With that said, what we did is um

we found a six different serving scenarios and each of these serving scenario we are pretty much aligned to specific user persona or application type and we are using these six engines that you can see uh on the screen for serving them out of data. bricks. The engines are SQL warehouse, API management, delta share, federation, OLTP and model serving. And what we are going to do now is we will go over each of these serving scenario in more details and we will talk about some best practices uh some tradeoffs that we had to do and what we learned down the road. And by the end of this presentation, our idea is for you if to to have like uh a blueprint which you can use for uh to implement those scenarios in your organization or your datab bricks environment. Okay. So let's start with the first one

and this is the AI and BI applications. Under this pattern, we have the BI tools, the Microsoft fabric integration and uh the datab bricks gen integration. If we start with the BI tools here, the first thing is we need to use the SQL warehouse. Uh that's needed because the SQL warehouse it's highly uh optimized for the SQL for any SQL workloads. So it's best to use it. Then we need to make one decision whether we want to use serverless or classic compute. If we select uh to use if we choose to use the classic compute um we have some benefits like we can uh do better on the network isolation part and also uh it's cheaper than the serverless option but there are also some downsides like um the starting time is around five minutes. So if you're running direct query reports, this means that you probably need to run your poster 24 by7 which could be really costly at the end of the day. From the other side, we have this uh serverless option which is uh which is costlier but the starting time is much faster uh and also it has all the latest and greatest features by data bricks out of the box. So based on our requirements, we need to choose which one is better for us. The next one and the next decision we need to make is if we want to scale out or if we want to scale up. Sometimes it's needed to scale both. But this is another another thing. The rule of the thumb here would be if

you're running very big workloads like you're processing a lot of data uh a big tables and you have big workloads we go with the scale up and then how we can make how we can see if we need to scale up our cluster. We can do this for example by going to the query history and run through our uh our our our queries and then there we can see if there is any data that's been spilled on the disk. If there is such data we can scale up uh from other side. If we are running direct query reports and if we are expecting hundreds of users hitting constantly the the warehouse then we need to go with the scale out option just because um each cluster in data bricks can handle up to 10 parallel queries only. The next one is the SQL driver

limitation currently and this is specifically for PowerBI. PowerBI driver um is producing some nonysql queries and it's nesting queries which is really really strange and data bricks is not very good at optimizing this. Uh so if you are creating a reports uh direct query reports you might you might see uh some strange queries that are being produced in datab bricks. So I know that database is currently working on uh fixing that. So we have so uh we hope the next couple of months this has been uh fixed. And then the caching and the cache warm up really important topic. Uh it's really essential for you uh to use the caching functionality because it speeds the processing a lot. But here there is a one thing that you need to know. The caching is per user. So if you're running a direct query report and you don't have the necessity for uh rowle security or column level masking then probably it's a better idea to use a common service account which is throwing all the queries so your queries can be cached uh at the end of the day. And the next one is the cache warm-up. It's a good idea to to preheat your cluster uh so when the users are opening the reports they are performing uh good and the first user do not like uh do not see the the first like caching uh queries but you need to know something here as well. It appears that if you're using a scale out cluster, the cache is not being replicated through the nodes. So if you have for example u a cluster with two nodes and the data is being cached on the first one uh if the cluster scales out to a second cluster to a second node and then a user comes in this means that the second user won't hit the cache and it will have uh not so good performance as the first one. The next integration we have in place is

Microsoft public integration. Uh here pretty much we have three options. The first is uh we can orchestrate with uh data factory and move the data from data bricks to uh fabric. The thing here is you need to make a data copy which is something that you need to decide uh if you are okay with. The second option is we can use fabric shortcuts or datab bricks mirroring functionality. But these are features uh in public preview. And also something really important they do not support uh the fabric mirror the data bricks mirroring does not support workspaces that are behind the private firewall uh behind the firewall. So this means that if if you want to build an enterprisegrade solution you can uh use this option. And the third option is data bricks writing to one leg. The idea here is you can move your goat layer uh from datab bricks to Microsoft fabric but I don't know if I can recommend you that the last option here for integration is datab bricks genie uh which is also utilizing SQL warehouse and the best practice here would be to create a separate SQL warehouse for all your uh genie workloads. This will ensure they have um enough bandwidth. They are not being throttled by other workloads and also it's easier for you to monitor the cost that's being produced by the genius uh by the Genie space. Okay. So, we spent a lot of time here on this slide but uh we put a lot of work together with our uh our like data engineering team and the guy and the team from datab bricks and Microsoft. So we thought it's a good idea to share all these insights with you guys. Going to the next integration here we

have the application integration which we are splitting into two categories. The first one is the high latency applications uh which are basically the all the applications that do not require millisecond performance and they are using typically JDBC, OBC or REST API. Uh here the main challenge was to see if we can um if datab brick can meet our serving requirements. So in order to do that what we did was um we decided to use Apache JMeter together with Azure W testing and to pretty much see how much uh what where are the limits of our SQL warehouses. So we tested different scenarios like we run uh 200 queries per second, we we run 500 users simultaneously etc. Uh so at the end of the day um we wanted to see how much how well it scales and as I said where are the limits. Um so also one other thing to ensure that we have a realistic tests we did the following we created a script which is pretty much shuffling different SQL queries and also shuffling different where statements so we ensure that we don't hit the cache and the results are realistic as possible. So the outcome was first the SQL warehouse was meeting our uh core requirements for data serving. The second one was something that we find for f found for JMeter and the problem with JMeter is it doesn't support Apache arrow. So the results from the tests are uh slower than the actual one. So we decided there to in order to meet our requirements we need to meet the the test without Apache error. This would mean that we would have some gap for errors and the the last one was for the large data set and the driver performance. It turns out that all the the results that are being produced if they are more than one megabyte they are being processed directly in datab bricks memory and they are super fast. But if the results are more than 1 megabyte then uh datab bricks creates like a chunk files and stores them in the DBFS and then the driver downloads those results from uh the DBFS.

uh the API management we decided instead of using uh datab bricks native REST API we created a wrapper on top which uses uh Asure API management. This ensures additional security and it's easier for the users uh to to hit the the REST API. They can just provide the warehouse ID and they're ready to go. The second option and the second category is our low latency applications and these are the applications uh which are our most critical ones and they require millisecond performance. Here uh currently we are using we are creating a copy inside a SQL server and we are having a radius cache layer on top and this ensures super fast performance. uh plus that is always there. It's available and it's ready to use. The downside here is we are creating additional data copy uh but you know you need to make a tradeoff uh in order to meet your requirements sometimes and the other option that we are currently exploring and I don't know how much we can talk about it that's data bricksp pretty much this should be a postgra instance uh inside datab bricks and the idea is to move the SQL and radius cache pattern inside datab bricks. So uh we can use all the goodies from uh unity catalog the common governance the common access control etc. The next pattern would be AI and ML

serving where we are currently exploring but we find out a few things that uh were worth mentioning. The first one is paper token versus provision throughut. If you're doing developments uh or you are testing some some some something, it's better to use the paper token option because it's cheaper uh and it's like you're paying only for what you're using basically. But once your model is production ready and you want to move it to production, then it's better to go to provision throughput. This will ensure you have uh reserve bandwidth and reserve query per seconds. And also one good op uh one important option with the provision throughput is to make sure you click on scale to zero option. This uh pretty much shuts down the end point 30 minutes after uh no one is like using it. So it saves a lot of cost. Also we are exploring the mosaic AI gateway the AI agent framework and the feature serving and we are trying to understand how they can help us with uh like checking what's going on what's the usage who is using the endpoints how we can create a better uh chat bots and tail lamps together with the API uh but the work here is currently in progress so I will skip to The next part because we are running out of time. The next

layer is data virtualization. And here basically the goal is the goal is uh we wanted to bring our existing data sources inside datab bricks. Um so not all of our data sources can be easily moved to datab bricks. So we had to have a way to serve it from there. So because at the end of the day we wanted this one unified serving layer that is available and it's managed only from one place instead of having like segregated uh systems. So to achieve this we decided to use um datab bricks federation capabilities and here uh we found out three important considerations that I want to share with you today. The first one is the limited available engines. So currently um there are not enough supported uh data sources. So you cannot connect your entire data data like landscape. The good thing though is datab bricks is currently working on that and they are constantly pushing new and new connectors. And today I was uh at one session where um the presenters show us the the road map and it seems like this issue will be fixed soon. The second one is the performance overhead and what I mean by that is pretty much your data lives in the source and in order to serve it out of data bricks it needs to create one additional data hub through data bricks. So if you are running a workloads that require very fast performance probably that's not the best idea for you and uh it's better to move your workloads to data bricks or as an alternative you can create a materialized view on top of this federated table which can sync the data uh and can ensure you have capacity to run. And the last one is the cost. Just because the queries are being run uh both on data bricks and on the like the database engine, it's generating cost on both ends. So it's good practice to monitor your system tables and monitor what's the cost that's being generated. Also this provides uh this promotes and supports the data the data mesh architecture where different teams can work on their uh preferred engines and they can expose the data through one and only serving layer which is unity catalog and also it can enhance the analytics with data with data without copying just because you are not storing anything in data bricks and you are hitting directly the the source and the endpoint.

The next one is uh the internal data sharing. And here our main idea is to have uh pretty much to leverage as much as possible the delta share capabilities. And to be more specific, we want to uh we are currently evaluating how we can use the private exchange. Um that's because we can use the private exchange to uh pretty much share data securely across the organization and also uh other people and other teams can discover this data from one single place and we are trying to see if we can implement this in our current data cataloging system. So we have everything on one place. Of course, this again supports the data product mindset because you can publish the entire data product within the within the the private exchange and other people can direct can like see the notebooks, the flows, the lineage, the data itself and everything could be stored in one place and also it eliminates the data silos and you do you don't do any copy across regions or organizations. The last one is the external sharing and here uh pretty much we want to enable the users to share data across organization in secured manner and this to be backed up by unity uh catalog governance and access control. uh we want to also exploring how we can uh utilize it to uh publish uh public data products and make them available for the wide audience. Uh but this one is currently uh in progress. So that's why I uh don't have much to share. So pretty much that was what I wanted to cover. So now I will pass again to Ivan. Okay. Thanks. So is so like it's not really an easy

thing uh to make a centralized data layer. But you know what we figured is that we need to make sure we cover I don't know 80 90% of the scenarios and do tailor solutioning for everything else. uh if you have your critical applications, sensitive applications, probably you don't want to run them out of a common serving scenario such as this one. But what you get is if you have your data available and discoverable and accessible in a centralized uh protected platform that is enabled for AI, ML and all these f futuristic uh and forwardlooking uh applications that you want to build then probably you can sacrifice here and there performance on an integration scenario. uh usually design should follow your requirements. So again there's no solution one solution to to fit them all. So you have to be opportunistic and of course most importantly be careful with vendor specific drivers. Sometimes there are some hidden things that you understand only uh after you go in production and start hitting performance challenges. The PowerBI thing, the Apacho thing. So all those things that you share and with that I think we have six more minutes. Uh this is all that we had to share and we'll open now the floor for questions. Uh if uh if you have a question please uh speak up on the mic.