- USPS leverages Databricks for a sharing-centric enterprise architecture.
- World Bank and Petrobrass share insights on overcoming data silos.
- The shift towards data products and agentic AI is accelerating innovation.
- Lakehouse Federation unifies governance across diverse data sources.
In an era where data is paramount and AI is rapidly reshaping industries, organizations face the dual challenge of securing highly sensitive data while simultaneously fostering an environment of open sharing and rapid innovation. This session, featuring insights from Databricks Solutions Architect Luke Bilbro and a panel including data leaders from the World Bank and Petrobrass, unveiled a blueprint for achieving unified governance and enterprise sharing for data and AI.
Luke Bilbro kicked off the session by highlighting the critical need for both robust governance and expansive sharing, particularly within public sector and research institutions. He emphasized that while data sensitivity demands stringent security, the imperative to modernize systems, cut costs, and leverage cutting-edge technologies like Generative AI necessitates seamless data exchange. The United States Postal Service (USPS) serves as a prime example, evolving from a focus on optimizing Spark workloads in 2020 to standardizing on a 'Unified Analytics Platform' (UAP) by 2023. A notable success story involved a new shipping fraud detection effort, where two independent teams, working in separate workspaces, were able to quickly combine critical data sets via Unity Catalog, enabling a SWAT team of data scientists to address the problem overnight.
Bilbro outlined three key lessons from USPS's journey: first, the ability to control data assets through both technology and a dedicated data administration team; second, the importance of building trusted, high-quality data products before widespread sharing; and third, standardizing on an open sharing framework. He stressed that simply cataloging technical metadata isn't enough; organizations must group assets into logically related 'data products' with clear ownership and rich metadata to ensure discoverability and usability. Unity Catalog acts as the 'executive branch' for data management, enforcing policies, while data strategy teams form the 'legislative branch' by defining governance frameworks.
The panel discussion, featuring Siraj Cotti from the World Bank and Marcelo Dotto from Petrobrass, further underscored these themes. Siraj Cotti detailed the World Bank's mission to alleviate poverty and how their data strategy, initially monolithic, now embraces a lakehouse framework to unify structured and unstructured content under a single governance layer. He highlighted the shift towards AI-driven workloads, self-service data products, and real-time analytics as key enablers post-Databricks implementation. Marcelo Dotto echoed the challenges of data silos at Petrobrass, explaining their move to a data mesh strategy with Databricks as its core. This decentralization empowered business units, significantly reducing the time to deliver data products from months to days, while maintaining governance through Unity Catalog.
Looking ahead, both panelists emphasized the transformative role of AI. Siraj Cotti warned against 'FOBO' (Fear of Being Obsolete) for organizations not embracing AI, advocating for agentic approaches and flexible, interoperable architectures. He stressed the need for a clear business strategy, defined value streams, and technology readiness to scale AI initiatives. Marcelo Dotto focused on AI's impact on data engineering, foreseeing automation of repetitive tasks and enhanced data observability and quality. The consensus was clear: a robust, open, and governed data foundation is not just beneficial, but essential for harnessing the full potential of AI and driving enterprise-wide innovation.
“If you just decentralize without governance, it will become a mess.”
- Luke Bilbro, Lead Solutions Architect, Databricks
