Revolutionizing Data Transformation: A Comprehensive Guide to Building an Autonomous Pipeline

As the keynote speaker at the Data Engineering Summit 2023, Rittika Jindal, a seasoned Cloud Architect at Thomson Reuters, enlightened the audience about the development of a dynamic and autonomous data transformation pipeline. The tech stack comprised AWS Fargate containers, DBT, and Airflow, focusing on the ‘T’ in the ETL or ELT pipeline. This pipeline equips businesses to construct their data marts independently, relying on SQL/DBT and creating dashboards for rapid insights without depending excessively on data engineers.

The Nature of the Transformation Pipeline:

Jindal emphasized the concept of a transformation pipeline, distinguishing it from conventional data extraction and loading pipelines. A transformation pipeline primarily targets the ‘T’ in ETL, assuming the data is already within the system and focusing solely on transformation. Thomson Reuters uses this pipeline to drive autonomy in their data analytics, incorporating technologies like DBT, Airflow, and AWS Fargate containers.

Promoting Autonomy in Data Analytics:

The primary goal of this transformation pipeline is to foster autonomy within Thomson Reuters’ data analytics team. Jindal explains that multiple teams operate in their data domain: the data engineering team, responsible for building data pipelines and maintaining years of data activity, and the analytics team, working closely with business units to generate insights, models, and dashboards. To bridge the gap between raw data and interpretable data for analysts and data scientists, Thomson Reuters devised the data transformation pipeline. This pipeline empowers the analytics team to transform data and build their marts, leveraging simple SQL queries.

Choosing the Right Tools: DBT and Airflow

After settling on the idea of a data transformation pipeline, the next challenge was choosing the right tools. After weighing multiple factors, including licensing costs, upskilling requirements, and ease of use, DBT emerged as the best fit for data wrangling and transformation. Open-source and user-friendly, DBT’s sole focus is on transforming the data already present in the system.

For workflow orchestration, Jindal and her team considered various tools, settling on Airflow. They needed a tool that could not only orchestrate DBT tasks but could also handle Python codes and third-party API calls. Airflow, as a platform that allows you to script your workflows using Python, complemented DBT well and fulfilled the requirements.

Implementing the Solution:

The final step was to integrate everything into a seamless architecture, adhering to the architectural principles of scalability and high availability. Thomson Reuters prioritizes building solutions on cloud and prefers serverless designs. This strategy led to a solution architecture utilizing Amazon ECR for storing Docker images and AWS Fargate for running serverless containers.

The architecture essentially works as follows: The Docker image containing DBT and Python is stored in the Amazon ECR. From there, Amazon ECS Task Definitions are created, which launch Fargate containers. Each container contains the DBT image, runs one model at a time, and then shuts down. The entire process is orchestrated through Airflow, making the system efficient, scalable, and high-performing.


Rittika Jindal’s talk shed light on an innovative approach to data transformation, emphasizing the importance of autonomy in data analytics and making intelligent choices in tool selection. By implementing a data transformation pipeline using DBT, Airflow, and AWS Fargate, Thomson Reuters has efficiently streamlined their data analytics process, minimizing dependencies and accelerating insights generation.

CDO Vision Dubai

26th October, 2023 | TAJ JUMEIRAH LAKES TOWERS | Dubai

Unite with Dubai's foremost Chief Data Officers at an exclusive networking event brought to you by AIM Leaders Council.

Our Latest Reports on Artificial Intelligence & Data Science

  • State of Global Capability Centers (GCCs) in India 2023

    The “GCC in India 2023” report offers a comprehensive examination of the rapidly evolving landscape of Global Capability Centers (GCCs) in India. It explores the different types of centers, including their functionalities and operational aspects. As businesses globally aim to centralize specific functions for better efficiency, India continues to be a preferred destination due to its talent pool and cost advantages.

  • Data Science Skills Study 2023

    In an era defined by the data revolution, the field of data analytics has become the backbone of decision-making across industries. As organizations strive to harness the power of data, the role of data and analytics professionals has evolved into one of paramount importance. The “Data Science Skill Study 2023” by AIM-Research delves into the multifaceted landscape of these professionals, shedding light on their skills, preferences, and the ever-evolving trends that shape their work.

  • Tackling the major roadblocks of text-based GenAI

    In recent years, the field of text-based generative artificial intelligence (AI) has witnessed remarkable advancements, revolutionizing natural language processing and generating human-like textual content. These AI models, such as GPT-3, have demonstrated unprecedented capabilities in generating coherent stories, answering questions, and even simulating human conversation.

    However, within this realm of immense promise, lie substantial challenges and obstacles that demand prudent navigation. As text-based generative AI achieves unprecedented capabilities, it simultaneously encounters complex roadblocks that necessitate careful consideration. These challenges encompass a range of intricate issues that span from accuracy and coherence to ethical considerations and contextual understanding.

    This report aims to explore and dissect the major roadblocks encountered in the domain of text-based generative AI and present effective strategies to overcome them.


  • Generative AI Tools: A Comprehensive Market Analysis

    The market for Generative AI tools is thriving, propelled by the expanding applications of these technologies and the growing recognition of their potential benefits. Industries across the spectrum, from tech and entertainment to healthcare and finance, are leveraging these tools to streamline processes, enhance creativity, and make strides in innovation.

    This report aims to provide an exhaustive analysis of Generative AI tools that are dedicated to individual functionalities. By investigating the market dynamics, uncovering trends, and identifying key players, this report offers essential insights into the current scenario and future prospects of these tools.


Subscribe to our Newsletter

By clicking the “Continue” button, you are agreeing to the AIM Terms of Use and Privacy Policy.

Supercharge your top goals and objectives to reach new heights of success!