It’s Actually Really Hard To Succeed With Data And AI Says Databricks CEO

“It’s actually really hard still to succeed with data and AI,” said Ali Ghodsi, co-founder and CEO of Databricks, during his keynote at the 2025 Data + AI Summit. “It’s a complexity nightmare of high costs and proprietary lock-in. It’s slowing down the organizations. We think this is the biggest problem.”

That challenge and Databricks’ proposed solution set the tone for one of the company’s most consequential summits to date. Over the course of 30 minutes, Ghodsi introduced a suite of new tools designed to simplify, accelerate, and govern AI adoption across the enterprise. But the real headline was unmistakable. Databricks is betting big on agents.

Agent Bricks

Databricks introduced Agent Bricks, a new product that automates the development and evaluation of AI agents using a company’s structured and synthetic data.But it’s not just a framework for spinning up task-specific bots but an attempt to answer the one question that enterprise AI is asking – how do we know these things are working?

“It doesn’t matter if the agent can crush programming contests or do Math Olympiad really well,” Ghodsi said. “We want it to do a specific job at the company. How do we know how it’s doing? That’s called evaluations or benchmarks.”

To that end, Agent Bricks uses large language models as automated “judges” tools that generate task-specific questions, expected answers, and rating metrics to score how well agents are performing. It’s an effort to bring transparency and rigor to what is still, in many organizations, a haphazard deployment process.

“If we can’t evaluate them, how do we even know? We’re going to unleash them in the workforce and not know how they’re doing, right? It could wreak havoc,” Ghodsi warned.

The Rise of Agent-Created Infrastructure

According to Ghodsi, the agentic era has already begun and it’s changing the very nature of databases.

“Now agents are creating databases,” he said. “Neon showed that 80% of their databases are created by agents, not by humans. Last year, that number was 30%. So in one year, they’ll be—probably 99% of the databases will be created by agents.”

Databricks acquired Neon Inc. in May to power its entry into the operational database space. The result is Lakebase, a managed Postgres database designed for AI-native applications. By building on Neon’s serverless architecture, Lakebase allows developers and increasingly agents to store and access dynamic data structures tied to real-time inference.

“We think this is going to be the future of all databases,” Ghodsi said. “Postgres has won. You really should properly separate compute and storage in that database using Postgres. It should be built for the AI era.”

While Postgres may be open, the move pits Databricks directly against Oracle, Microsoft, and Amazon in the operational database market which is traditionally dominated by proprietary systems and decades-old vendor relationships. “This technology hasn’t changed for 40 years,” Ghodsi noted. “And it’s so locked in. Once you have your data there, you can’t move it out.”

From Data Warehousing to Data Intelligence

The agent push is just one layer of Databricks’ broader shift toward what it calls data intelligence infusing AI throughout the platform, from data governance to user experience.

On one side of the strategy is Genie, an AI-powered natural language interface used by 81% of Databricks’ customer base. Users create “data rooms” in Unity Catalog, then query data using plain English. Genie dynamically assembles agents to handle the query, write code, and deliver results. On the other side is the Databricks Assistant, now used by 98% of customers. It’s designed to understand not just syntax, but the company’s internal data, structure, and business logic.

“We want to democratize access to data,” Ghodsi said. “In the past, you had to know Python or SQL to get value. We want to challenge that. It should be enough if you just speak English or any language.”

Databricks isn’t the only one racing toward this “no-code AI” future, but it’s putting its weight behind making data more accessible and usable. Lakeflow Designer, another product unveiled at the summit, allows users to build production data pipelines using drag-and-drop tools and a generative AI assistant. Ghodsi described it as “vibe designing”—a reference to the growing trend of abstracting technical complexity behind natural language or visual interaction.

“It’s kind of like vibe coding if you are not coding at all,” he said. “It’s a new market for us, a new market opportunity.”

The Governance Layer No One Talks About

But amidst the demos and feature rollouts, Ghodsi returned repeatedly to one recurring pain point which is governance.

Databricks has long argued that complexity in enterprise data architecture stems not just from fragmentation but from fragmented security models and metadata. “Each system has its own access control and governance,” he explained. “And that’s what’s making it really, really hard.”

That’s why the company is pushing hard on Unity Catalog, which Ghodsi positioned as the only governance solution that spans structured data, unstructured data, dashboards, and AI models—all from a centralized layer. “No governance solution today out there, outside of Unity Catalog, really does it for all your data assets,” he said.

He also criticized narrow interpretations of governance in the market. “Despite all the talk, most focus is just on access control on structured data. Efforts like Polaris are focused on a narrow slice,” he said. “We need unified governance.”

The company also reaffirmed its support for open formats like Delta Lake and Iceberg, citing over 2 billion downloads of Apache Spark and 1 billion of Delta Lake. With Unity Catalog now supporting both Delta and Iceberg natively and open-source implementations of Hive Metastore and Iceberg REST Catalog, Databricks is leaning further into open standards as a wedge against proprietary lock-in.

A $100M Talent Push, and a Warning

Technology aside, Databricks also announced the launch of Databricks Free Edition and a $100 million investment in AI and data education. The goal is to provide universal access to its platform and self-paced Databricks Academy courses, without requiring corporate emails or credit cards.

“Everyone we speak to is constrained by the same problem: not enough people with the right data and AI skills,” Ghodsi said. “With the right investment and the right tools, we will see more AI innovation in this next generation than all previous generations combined.”

It’s a timely message. According to the U.S. Bureau of Labor Statistics, data scientists are among the fastest-growing professions in the U.S. Meanwhile, 80% of executives surveyed by the World Economic Forum expect AI to fundamentally reshape their organizations by 2030.

But Ghodsi also issued a warning to industry leaders. “If you don’t know what your agents are doing, it’s dangerous,” he said. “You’re flying blind.”

Talent Wars and the AI Exodus

Behind the platform talk was also a candid admission on the AI labor market is more volatile than ever. Ghodsi described a shifting landscape where top researchers are fleeing companies en masse due to internal “drama,” leaving startups and incumbents scrambling for top-tier talent.

“It’s very dynamic. We’ll see a popular company, there’s drama and then everybody leaves,” he said. “I’ve not seen it in the last 15 years. Suddenly, everybody wants to leave one company, then join another. It just goes back and forth.”

It’s a labor market that Ghodsi believes will favor startups that can move fast, build deep agent infrastructure, and offer a mission-led alternative to big tech.

“What’s our secret sauce?” Ghodsi asked. “We have the customers, and we have the data. That’s the differentiator.”