In a recent talk at the Data Engineering Summit 2023 in Bangalore, a conference by AIM, Sunil Krishnareddy, VP & Head of Data Engineering services at Genpact, shed light on the evolving landscape of data engineering in the era of Artificial Intelligence (AI). With a career spanning 25 years, starting as a software engineer with Infosys and leading application development practices for global Hi-tech customers, Sunil has a proven track record of delivering digital transformation programs for the world’s Fortune 500 companies.

The Evolution of Data Engineering

Sunil began his talk by emphasizing the importance of data in the current age. He quoted historian Yuval Noah Harari, who suggested that in the future, the only contribution of human beings will be their own data. This statement underscores the significance of data and its potential to drive AI and machine learning advancements.

Sunil also highlighted the rapid pace of technological change, with hundreds of AI tools emerging daily. He stressed that the success of AI is largely dependent on good data engineering.

Data-Driven vs Value-Driven Organizations

One of the key topics Sunil discussed was the difference between data-driven and value-driven organizations. He explained that while technology allows organizations to access and analyze vast amounts of data, the real challenge lies in deriving value from this data.

Sunil argued that data in itself has no value and is, in fact, a cost to produce, store, transform, and consume. The real value of data lies in its ability to drive business outcomes. He encouraged organizations to start from their business outcomes and work backwards to see how data can assist in achieving these outcomes.

Data Mesh vs Data Fabric

Sunil also touched upon the debate between adopting a data mesh or a data fabric approach to data architecture. He explained that a data mesh is a decentralized approach where governance is decentralized and data is provided as a set of data products. On the other hand, a data fabric is a more centralized approach where data consumers and producers interact through a middle layer.

He suggested that neither approach is inherently superior; the choice depends on an organization’s specific needs. However, he noted that many organizations are settling on a middle path that combines elements of both approaches.

Data Lakes, Data Warehouses, and Streams

Sunil discussed the convergence of data lakes, data warehouses, and streams. He noted that while traditionally, data warehouses were suited for structured data and data lakes for unstructured data, modern software can handle both. Similarly, while these systems were traditionally suited for batch data, modern software can also handle real-time data streams.

The Importance of Metadata and Data Catalogs

Sunil emphasized the importance of metadata and data catalogs in understanding and managing data. He likened data catalogs to the labels on bottles, without which one wouldn’t know what they’re drinking. Similarly, without a data catalog, organizations are “flying blind,” unable to fully understand or utilize their data.

He also highlighted the importance of data lineage, which shows how data flows through an organization. This can be crucial for understanding how data transforms as it moves through different systems and for maintaining data quality.

Key Insights from the Talk

• The success of AI is largely dependent on good data engineering.
• Data in itself has no value; its value lies in its ability to drive business outcomes.
• The choice between a data mesh or a data fabric approach to data architecture depends on an organization’s specific needs.
• Modern software can handle both structured and unstructured data, as well as batch and real-time data streams.
• Metadata and data catalogs are crucial for understanding and managing data.

Sunil’s talk provided valuable insights into the evolving landscape of data engineering and underscored the importance of understanding and managing data effectively in the age of AI. His insights serve as a guide for organizations navigating the complex world of data engineering.