Search
Close this search box.

Data Engineering in the New Age: AI is Here, Where is Your Data?

"Data engineering is the cornerstone of AI success, and its value lies in its ability to drive business outcomes," says Sunil Krishnareddy, VP & Head of Data Engineering services at Genpact.

In a recent talk at the Data Engineering Summit 2023 in Bangalore, a conference by AIM, Sunil Krishnareddy, VP & Head of Data Engineering services at Genpact, shed light on the evolving landscape of data engineering in the era of Artificial Intelligence (AI). With a career spanning 25 years, starting as a software engineer with Infosys and leading application development practices for global Hi-tech customers, Sunil has a proven track record of delivering digital transformation programs for the world’s Fortune 500 companies.

The Evolution of Data Engineering

Sunil began his talk by emphasizing the importance of data in the current age. He quoted historian Yuval Noah Harari, who suggested that in the future, the only contribution of human beings will be their own data. This statement underscores the significance of data and its potential to drive AI and machine learning advancements.

Sunil also highlighted the rapid pace of technological change, with hundreds of AI tools emerging daily. He stressed that the success of AI is largely dependent on good data engineering.

Data-Driven vs Value-Driven Organizations

One of the key topics Sunil discussed was the difference between data-driven and value-driven organizations. He explained that while technology allows organizations to access and analyze vast amounts of data, the real challenge lies in deriving value from this data.

Sunil argued that data in itself has no value and is, in fact, a cost to produce, store, transform, and consume. The real value of data lies in its ability to drive business outcomes. He encouraged organizations to start from their business outcomes and work backwards to see how data can assist in achieving these outcomes.

Data Mesh vs Data Fabric

Sunil also touched upon the debate between adopting a data mesh or a data fabric approach to data architecture. He explained that a data mesh is a decentralized approach where governance is decentralized and data is provided as a set of data products. On the other hand, a data fabric is a more centralized approach where data consumers and producers interact through a middle layer.

He suggested that neither approach is inherently superior; the choice depends on an organization’s specific needs. However, he noted that many organizations are settling on a middle path that combines elements of both approaches.

Data Lakes, Data Warehouses, and Streams

Sunil discussed the convergence of data lakes, data warehouses, and streams. He noted that while traditionally, data warehouses were suited for structured data and data lakes for unstructured data, modern software can handle both. Similarly, while these systems were traditionally suited for batch data, modern software can also handle real-time data streams.

The Importance of Metadata and Data Catalogs

Sunil emphasized the importance of metadata and data catalogs in understanding and managing data. He likened data catalogs to the labels on bottles, without which one wouldn’t know what they’re drinking. Similarly, without a data catalog, organizations are “flying blind,” unable to fully understand or utilize their data.

He also highlighted the importance of data lineage, which shows how data flows through an organization. This can be crucial for understanding how data transforms as it moves through different systems and for maintaining data quality.

Key Insights from the Talk

  • The success of AI is largely dependent on good data engineering.
  • Data in itself has no value; its value lies in its ability to drive business outcomes.
  • The choice between a data mesh or a data fabric approach to data architecture depends on an organization’s specific needs.
  • Modern software can handle both structured and unstructured data, as well as batch and real-time data streams.
  • Metadata and data catalogs are crucial for understanding and managing data.

Sunil’s talk provided valuable insights into the evolving landscape of data engineering and underscored the importance of understanding and managing data effectively in the age of AI. His insights serve as a guide for organizations navigating the complex world of data engineering.

Picture of AIM Research
AIM Research
AIM Research is the world's leading media and analyst firm dedicated to advancements and innovations in Artificial Intelligence. Reach out to us at info@aimresearch.co
Subscribe to our Latest Insights
By clicking the “Continue” button, you are agreeing to the AIM Media Terms of Use and Privacy Policy.
Recognitions & Lists
Discover, Apply, and Contribute on Noteworthy Awards and Surveys from AIM
AIM Leaders Council
An invitation-only forum of senior executives in the Data Science and AI industry.
Stay Current with our In-Depth Insights
The Biggest Exclusive Gathering Of CDOs & Analytics Leaders In United States

MachineCon 2024
26 July 2024, New York

MachineCon 2024
Meet 100 Most Influential AI Leaders in USA
Our Latest Reports on AI Industry
Supercharge your top goals and objectives to reach new heights of success!

Cutting Edge Analysis and Trends for USA's AI Industry

Subscribe to our Newsletter