Peter Ngigi

- Refactored large scale ETL pipeline to run end to end in 50% less time by rearchitecting better data models, speeding up API authentication and parallelization of ingestion processes

• Created custom python scripts to automate critical processes such as creation of IAM users, EMR clusters and AWS Glue schemas as part of a project to migrate 300 index builders from an on-premises Hadoop cluster to AWS EMR.
• Coordinated with owners of ETL pipelines as well as internal libraries to modify them for deployment in AWS, as well as iteratively debug bottlenecks to the migration process.
• Led the engineering for a generalized solution to migrate large artifacts utilized by multiple ETL jobs from HDFS to AWS S3, while guaranteeing no adverse impact to performance or processing time.

• Successfully migrated over 100 ETL pipelines from an old Hadoop cluster to a new cluster by leveraging a Python library named Skein and Airflow to submit jobs to Hadoop Yarn.
• Implemented data serialization to Hive using PyArrow and seamless writes to and reads from HDFS and Hive using Ibis/HDFSCLI.
• Ensured no adverse impact to model metrics by conducting data comparisons and validation, including row aggregate counts for thousands of Hive tables and machine learning model outputs.