Data Engineering in the New Age: AI is Here, Where is Your Data?

"Data engineering is the cornerstone of AI success, and its value lies in its ability to drive business outcomes," says Sunil Krishnareddy, VP & Head of Data Engineering services at Genpact.

In a recent talk at the Data Engineering Summit 2023 in Bangalore, a conference by AIM, Sunil Krishnareddy, VP & Head of Data Engineering services at Genpact, shed light on the evolving landscape of data engineering in the era of Artificial Intelligence (AI). With a career spanning 25 years, starting as a software engineer with Infosys and leading application development practices for global Hi-tech customers, Sunil has a proven track record of delivering digital transformation programs for the world’s Fortune 500 companies.

The Evolution of Data Engineering

Sunil began his talk by emphasizing the importance of data in the current age. He quoted historian Yuval Noah Harari, who suggested that in the future, the only contribution of human beings will be their own data. This statement underscores the significance of data and its potential to drive AI and machine learning advancements.

Sunil also highlighted the rapid pace of technological change, with hundreds of AI tools emerging daily. He stressed that the success of AI is largely dependent on good data engineering.

Data-Driven vs Value-Driven Organizations

One of the key topics Sunil discussed was the difference between data-driven and value-driven organizations. He explained that while technology allows organizations to access and analyze vast amounts of data, the real challenge lies in deriving value from this data.

Sunil argued that data in itself has no value and is, in fact, a cost to produce, store, transform, and consume. The real value of data lies in its ability to drive business outcomes. He encouraged organizations to start from their business outcomes and work backwards to see how data can assist in achieving these outcomes.

Data Mesh vs Data Fabric

Sunil also touched upon the debate between adopting a data mesh or a data fabric approach to data architecture. He explained that a data mesh is a decentralized approach where governance is decentralized and data is provided as a set of data products. On the other hand, a data fabric is a more centralized approach where data consumers and producers interact through a middle layer.

He suggested that neither approach is inherently superior; the choice depends on an organization’s specific needs. However, he noted that many organizations are settling on a middle path that combines elements of both approaches.

Data Lakes, Data Warehouses, and Streams

Sunil discussed the convergence of data lakes, data warehouses, and streams. He noted that while traditionally, data warehouses were suited for structured data and data lakes for unstructured data, modern software can handle both. Similarly, while these systems were traditionally suited for batch data, modern software can also handle real-time data streams.

The Importance of Metadata and Data Catalogs

Sunil emphasized the importance of metadata and data catalogs in understanding and managing data. He likened data catalogs to the labels on bottles, without which one wouldn’t know what they’re drinking. Similarly, without a data catalog, organizations are “flying blind,” unable to fully understand or utilize their data.

He also highlighted the importance of data lineage, which shows how data flows through an organization. This can be crucial for understanding how data transforms as it moves through different systems and for maintaining data quality.

Key Insights from the Talk

  • The success of AI is largely dependent on good data engineering.
  • Data in itself has no value; its value lies in its ability to drive business outcomes.
  • The choice between a data mesh or a data fabric approach to data architecture depends on an organization’s specific needs.
  • Modern software can handle both structured and unstructured data, as well as batch and real-time data streams.
  • Metadata and data catalogs are crucial for understanding and managing data.

Sunil’s talk provided valuable insights into the evolving landscape of data engineering and underscored the importance of understanding and managing data effectively in the age of AI. His insights serve as a guide for organizations navigating the complex world of data engineering.

CDO Vision Dubai

26th October, 2023 | TAJ JUMEIRAH LAKES TOWERS | Dubai

Unite with Dubai's foremost Chief Data Officers at an exclusive networking event brought to you by AIM Leaders Council.

Our Latest Reports on Artificial Intelligence & Data Science

  • State of Global Capability Centers (GCCs) in India 2023

    The “GCC in India 2023” report offers a comprehensive examination of the rapidly evolving landscape of Global Capability Centers (GCCs) in India. It explores the different types of centers, including their functionalities and operational aspects. As businesses globally aim to centralize specific functions for better efficiency, India continues to be a preferred destination due to its talent pool and cost advantages.

  • Data Science Skills Study 2023

    In an era defined by the data revolution, the field of data analytics has become the backbone of decision-making across industries. As organizations strive to harness the power of data, the role of data and analytics professionals has evolved into one of paramount importance. The “Data Science Skill Study 2023” by AIM-Research delves into the multifaceted landscape of these professionals, shedding light on their skills, preferences, and the ever-evolving trends that shape their work.

  • Tackling the major roadblocks of text-based GenAI

    In recent years, the field of text-based generative artificial intelligence (AI) has witnessed remarkable advancements, revolutionizing natural language processing and generating human-like textual content. These AI models, such as GPT-3, have demonstrated unprecedented capabilities in generating coherent stories, answering questions, and even simulating human conversation.

    However, within this realm of immense promise, lie substantial challenges and obstacles that demand prudent navigation. As text-based generative AI achieves unprecedented capabilities, it simultaneously encounters complex roadblocks that necessitate careful consideration. These challenges encompass a range of intricate issues that span from accuracy and coherence to ethical considerations and contextual understanding.

    This report aims to explore and dissect the major roadblocks encountered in the domain of text-based generative AI and present effective strategies to overcome them.


  • Generative AI Tools: A Comprehensive Market Analysis

    The market for Generative AI tools is thriving, propelled by the expanding applications of these technologies and the growing recognition of their potential benefits. Industries across the spectrum, from tech and entertainment to healthcare and finance, are leveraging these tools to streamline processes, enhance creativity, and make strides in innovation.

    This report aims to provide an exhaustive analysis of Generative AI tools that are dedicated to individual functionalities. By investigating the market dynamics, uncovering trends, and identifying key players, this report offers essential insights into the current scenario and future prospects of these tools.


Subscribe to our Newsletter

By clicking the “Continue” button, you are agreeing to the AIM Terms of Use and Privacy Policy.

Supercharge your top goals and objectives to reach new heights of success!