Abhishek Agrawal

ETL Pipeline for End to End Specials Use Case Squad Spearheading the migration process from Hadoop cluster to Azure Databricks spark cluster. Ensuring optimal code performance in the spark cluster and constantly seeking out ways to improve it. Industrializing the code base to facilitate seamless scaling. Taking the lead in forming the CI/CD process for both Azure Data Factory and Azure Databricks. Collaborating closely with Data Scientists to provide them with accurate data to derive actionable business insights. RETAIL HUB DASHBOARD Developed a dynamic spark notebook to pivot the web crawling data efficiently. Integrated Azure Synapse Warehouse and Data Lake with ADF and created parameterised pipeline in Data Factory for an end-to end ETL process. Designing and developing Warehouse Objects and Stored Procedures for streamlined data processing. Developing a Logic app to send daily email

EPM ONYX BUSINESS REPORTING 1)Gathered requirements from stakeholders for Business Reporting. 2)Prepared Technical & Scope Analysis document for Data Models which includes Facts & Dimensions mapping. 3)Fetched Data from a Multidimensional source system(EPM) using API through Azure Data Factory to Built ETL Pipeline. 4)Built Full load and Delta load Pipeline in Azure Data Factory. 5)tored the raw data into Azure Data Lake in the Date Time folder structure. 6)Written transformation logics using Pyspark and SparkSQL in Databricks to convert the raw data into Facts and Dimensions. 7)Done Erwin Modelling using Erwin Studio and Implemented Semantic Data Model using Azure Analysis Services for various Analytics Power BI reports/dashboards. 8)Created a technical incident/challenge Document for effective communication across the team. MLT O2C DASHBOARD 1)Added a new report in existing order to the cash dashboard to track Market Lead Time(MLT). 2)Read the uploaded data through Logic App into Databricks from Azure Blob Storage. 3)Made required transformation in Databricks using SQL and Python and written the transformed data into curated layer of Azure Data Lake Gen2. 4)Read the Data into Azure Synapse and done Erwin Modelling using ER Studio and implemented Data Models in Azure Analysis Service for Power BI Visualization. Disaster Recovery and CI/CD Pipeline 1)Taken backups for Azure Databricks, Azure Data Factory, Azure Analysis Service using Databricks Command-Line(CLI), ARM templates, SSMS, and PowerShell. 2)Designed a CI/CD pipeline using Azure DevOps to automate the build and release processes across various environments for Azure Data Factory (ADF), Azure Synapse (Data Warehouse) & Azure Analysis Services.

AUTOMOTIVE MIS DASHBORAD REPORTING USING AZURE 1)Built a data integration ETL pipeline from SAP S/4HANA and Dynamics 365 source systems and moved data from on premise to cloud with the help of Azure Data Factory. 2)Created date hierarchy partition in Azure Data Lake Storage Gen 2 to store the data in multiple layers. 3)Transformed data using Azure Synapse (Azure Data Warehouse) created Schema, Facts, and Dimensions using MS SQL to organize and populate the data in table structures for various source systems with Stored Procedures and created a semantic layer for data modeling in Azure Analysis Services to build and deploy multiple interactive analytics dashboards or reports using Power BI. 4)Handled Dynamics 365 System End to End from Source Integration to making reports in Power BI. KPI REPORTING USING AZURE 1)Migrating data from On Premise server to Azure Data Lake using Azure Data Factory and processing CSV, Json, XML file using Scala, PySpark, Spark SQL in Databricks. 2)Writing processed file to Azure SQL and Delta Lake using Databricks and moving it to Archive container in Databricks. 3)Building Dashboard on top of Delta Table and Orchestrating the whole ETL process using Azure Data Factory. AUTOMATED HYPERANALYTICS PLATFORM 1)Developed an internal Data Science Platform using Python and ML Algorithms which can automate the various Data Science processes thereby reducing the computational complexity and time required to solve a specific problem. 2)Used Apache Airflow, Pyod Library, Facets, Pixiedust library, Auto Keras, H20 and ML Algorithms etc for overall automation. 3)Utilized the Flask module for developing the backend machine learning framework.