Data Engineer
Pinterest
2021-02-01-Present
Working with data in the advertising team.
Experience with designing, building, delivery, and maintenance of accurate large scale data p...
Working with data in the advertising team.
Experience with designing, building, delivery, and maintenance of accurate large scale data processing backends
- Development of API’s to integrate with external vendors
- Creation and maintenance of Spark scripts for large data processing
- Experience in creating screens using Angular 2
- Creation and maintenance of workflows using Airflow
- Experience in projects focused on privacy of customer information
Java
Jenkins
Python 3
View more
Java
Jenkins
Python 3
Apache Spark
Apache Hadoop
Apache Airflow
AWS
View more
Technical leader
IHM Stefanini
2020-02-01-2021-02-01
Experience with processing large volumes of data (Big Data).
- Leader of the Data Lake construction project using AWS solutions.
- Design...
Experience with processing large volumes of data (Big Data).
- Leader of the Data Lake construction project using AWS solutions.
- Design of a Data Lake going through the steps of data ingestion, storage, processing, cataloging and
visualization.
- Experience with AWS (IAM, Cognito, EC2, API Gateway, CloudWatch, SQS, SQN, Kinesis Data
Streams, Kinesis Firehose, Elasticsearch Service, S3, Lambda, Glacier, Glue, Athena, RedShift,
Redshift spectrum, EMR, QuickSight).
- Use of relational (Postgresql) and non-relational (DynamoDB) databases.
- Cost estimation for projects using AWS solutions.
- Use of OpenAPI (OA3) to create the data consumption API for Data Lake.
- Knowledge of cloud infrastructure provisioning tools via code such as: Terraform.
- Creation of a cluster with 5 machines to support JupterHub on AWS (EMR).
- For developing knowledge in Nodejs, Python and Java 8.
- Version control of the generated code using Git, Git Flow and GitLab.
- Communication and interpretation of problems with the business areas.
Git
GitLab
Big Data
View more
Git
GitLab
Big Data
Scrum
Apache Spark
Apache hudi
AWS
View more
Software Architect
IHM Stefanini
2019-02-01-2020-01-01
- Architect in the innovation sector, including the areas of systems development and data science.
- Constant contact with the company's ...
- Architect in the innovation sector, including the areas of systems development and data science.
- Constant contact with the company's POs to survey the requirements to be developed.
- Defining the architecture of systems that need a solution in real time and almost in real time using
Java 8 and Nodejs on AWS (IAM, EC2, S3, API Gateway, AWS Lambda, CloudWatch, SQS, SQN,
RDS).
- Survey of requirements and implementation of SaaS systems using the Serverless Framework.
- Definition in the construction of micro services and use of Docker and Kubernetes.
- Use of Apache Kafka (topics, producers, consumers) and use of the AWS MSK solution.
- Using BDD concept for software development in the team together with the Cucumber solution.
- Architectural definition of systems using Python, Nodejs and Java 8.
- Mapping / Documenting the company's systems using tools such as Draw.io and Cloudcraft.
- Participation in the architectural definition of new products.
Python
Java
Django
View more
Python
Java
Django
Node.js
PostgreSQL
Cucumber
Docker
Apache Kafka
Serverless
AWS
View more