Sudeep Gupta

Airflow Platform

• Developed a self-hosted Airflow Platform for Farfetch with integrated security (customisedRBAC logins, optimised Secrets Backend), remote logging, observability, testing, build pipelines and automated redeployment. Automated and modularised the entire process with Terraform and GitOps (with ArgoCD) for deployment.
• Successfully migrated data teams from Google Composer effectively reducing the cloud cost, and increased scalability on shared infrastructure
• Custom Secret Backend which improves dags parsing time by 100x and makes the process more reliable and fault tolerant
• Open Source contributions to improve the upstream software •

Observability

• Prometheus High Availability Setup: Integrated Prometheus with DeadMansSnitch to ensure this was always available, and retired the Azure Container Insights solutions, effectively saving 30000 USD per month across 3 Data Centers
• SetupthealertingintegrationofvariousserviceswithPagerDuty,anddevelopmentofenhanced alerting rules to support various applications on the platform
• Creation of Grafana Dashboards and Runbooks to monitor & action on incidents for various Platform components

• Various Infrastructure Optimisations
Various initiatives within the ambit of Data Platform to reduce the infrastructure cost

• Enabling Spot Machines across several applications deployed on K8s, and Databricksworkloads which helped save more than 100,000 USD annually
• Databricks Overwatch was deployed for Governance, and this data was used to track potentialwastage of compute. This initiative helps save 120,000 USD annually

• Data Fabric for Equity ResearchThere was a need to store data from various sources – both structured, and semi structured for the purpose of research and analysis by Active Equity groups for signal generation. The entire system was automated and could be deployed with optional add-on components. Technology Stack: Google Cloud Platform, MongoDB, Ansible, Python (flask)
• Migration to Cloud, Kubernetes and Airflow
  The entire data infrastructure was on-prem and hence, lagging behind in terms of scalability and provisioning of new technological components. I was the lead developer on this project
  Migrated core product application which was used to model various Mortgage Assets data tomake it cloud ready
  Automated and migrated existing business logic and data pipelines to Airflow (GoogleComposer), and reduced the execution time from 2 days to 10 hours.
  Decoupled execution details from the business logic of the pipelines by developing adeclarative/state based extension to the Airflow pipeline
  Setup a small Airflow cluster on an on-prem OpenShift Cluster for various internal POCswhich weren’t fully ready for cloud
  Containerised various services/applications and deployed them on K8s
  Technology Stack: Google Cloud, OpenShift, Kubernetes, Docker, Airflow, Scala, Python
• Low Latency/Interactive Analytics Data Lake Platform
  Designed and developed a data lake platform which would support low latency/interactive analytics and enable modelers to focus on data.
  ◦ Interacted with various holders to understand the computation needs (system loads, datagrowth, computation effort type- batch, ad-hoc queries, report generation etc.)
  Structured and codified the approach to storing data in the Data Lake to prevent it frombecoming a Swamp
  Technology Stack: Hadoop, Hive, Spark, PrestoDB, Drill, Python