- Federal CIOs prioritize AI, ML, and cloud for evidence-based decisions.
- Deloitte estimates $41 billion in annual savings from data-driven automation.
- Lakehouse architecture simplifies complex data pipelines and enhances performance.
- Transparency through public dashboards drives accountability and addresses disparities.
The public sector is undergoing a profound transformation, driven by an urgent need to leverage data and artificial intelligence for better citizen services, enhanced national security, and operational efficiency. Leaders from agencies like the FBI, Department of Defense (DoD), Department of Veterans Affairs (VA), and Hennepin County shared their journeys at the Data + AI Summit 2021, highlighting both the immense potential and the significant challenges in this digital evolution.
Howard Levinson, General Manager for Databricks Federal, set the stage by emphasizing the White House's prioritization of data and AI. With over a billion dollars invested in AI research in 2021 and federal CIOs ranking AI, ML, and cloud as top priorities, the drive for data-driven automation is clear. Deloitte estimates these initiatives could yield an astounding $41 billion in annual savings for the government.
However, this journey is not without its hurdles. Speakers identified a critical skills gap, the daunting task of migrating 12,000 on-premise data centers to the cloud, and the challenge of federating "islands of data" to enable advanced analytics. John Larson, Senior Vice President for Analytics at Booz Allen Hamilton, underscored that while AI is relatively easy in a sterile lab environment, scaling it to production with massive, messy data, expensive compute resources, and complex business processes often leads to failure. He stressed the importance of cultural, data, technology, and process readiness.
Depali Pawali, a Data Science Lead at the FBI, showcased Project Prometheus, their content warehouse that leveraged Databricks to simplify complex data ingest pipelines. By minimizing reliance on NoSQL solutions and adopting a Delta Lakehouse implementation, they significantly improved performance, reducing job run times from hours to minutes. Similarly, Dave Fuller from the VA's Data Analytics Division recounted how their CFO dashboard, which once took five days for ETL processes, now completes in a couple of hours, enabling deeper financial insights over eight years of data instead of just two. Hennepin County's Andrew Lucius highlighted their use of public Power BI dashboards for transparency in the county attorney's office, revealing case demographics and racial disparities to drive targeted programs. He also shared how machine learning predicts property tax delinquencies to intervene earlier and support vulnerable populations.
Paul Donahoe, a Senior Technical Advisor at CMS, provided a historical perspective, noting that the biggest change in his 35-year career has been the separation of compute and storage in the cloud, enabling multiple teams to access data without performance degradation. Yet, he emphasized that the biggest challenge remains cultural—overcoming the ingrained habit of data hoarding. Andrew Lucius echoed this, stressing the importance of building relationships and trust to convince different departments to share data, rather than imposing top-down mandates. The consensus was that while technology provides the tools, successful data transformation hinges on organizational readiness, collaboration, and a shared vision.
“Data is the new ammunition in our arsenal that improves decision making and connects the DoD's boardroom and battlespace.”
- Howard Levenson, GM, Area Vice President, Databricks Federal, LLC
