I worked as a Software Engineer and as a Data Engineer on a Data Platform team at a company that stored roughly 10% of all electronic invoices in Brazil. That was billions of records in our databases, and everyday it received tens of millions more. My team was responsible for providing APIs for other engineering teams, both to ingest new records into our platform and to retrieve records based on various filters. These APIs enabled the engineering teams to develop new features for end-users without worrying about business logic, which database the data was stored in, data migrations, data consistency, etc.
As a software engineer, I've developed event-driven microservices for real-time data processing and HTTP APIs. My day to day tasks included:
As a data engineer, I've built batch pipelines to migrate billions of records from one database to another in just a few hours and I would also maintain several Airflow DAGs that triggered ETL pipelines that populated data marts in our data warehouse. I was also Data Engineering Chapter Leader, responsible for tutoring junior data engineers, creating data engineering trainings, diffusing good practices and standards for creating pipelines and organize meetings to discuss technology and tools within the Data Engineering Scope.
More recently I was being trained to become a Tech Lead. I took over my Tech Lead's responsabilites when he went on vacation for 20 days, which included talking to our manager to decide which projects to prioritize, help the team plan the weekly tasks, resolve major incidents and plan the next quarter roadmap.
• Responsible for a team of three people and several contractors; the company provides a platform with editable and ready to use conte...
• Responsible for a team of three people and several contractors; the company provides a platform with editable and ready to use content for psychologists to post on their Instagram. Our product return rate is less than 2%.
• Managed two projects that replaced several products with only one that was simpler, more intuitive to use and that increased in 50% the company return on investment.
• Created and optimized all social media strategy campaigns on Facebook Ads.
• Wrote the copy for all product launches including e-mails, Whatsapp/Telegram messages, and social media posts. Sold 56% more than expected on the first day and 30% more on the first week.
• Developed distributed event-driven microservices for real-time processing large volumes of data; these workers backed the most criti...
• Developed distributed event-driven microservices for real-time processing large volumes of data; these workers backed the most critical services of the company and handled 10 million records each per day. We believe in the motto “you build it, your unit, you own it”: all systems were monitored, had dashboards, alerts and a CI/CD pipeline.
• Built batch pipelines to migrate data between databases with billions of records within 5 days, including data extraction from different sources, data transformation, data write, data audit and new system up and running.
• Developed HTTP APIs to retrieve data from internal databases and serve it to stream align teams. The mission of my team, the Electronic Invoice Platform, was to enable other engineering teams to develop new features for the end user without the need to worry about business logic regarding the different types of documents or the technology needed to store or retrieve them.
• Devised and developed an internal library in Go to be used in our workers development to read and write from Bigtable, encode and decode data in Wire Format/Avro, generate Change Data Capture, and send events to Kafka. This library reduced in 75% the time to develop, test and deploy a new system.
• Provided maintenance for batch and streaming pipelines, modeled data from different storages and data marts, orchestrated hourly, daily and weekly DAGs for ETL pipelines.
• Data Engineering Chapter leader, a position that impacts all data engineers, analytics engineers, BI engineers and some software engineers in the company. My responsibilities included tutoring Junior Data Engineers, creating data engineering trainings, diffusing good practices and standards for batch pipelines, and organize recurrent meetings to discuss technology and tools within the Data Engineering scope.
• Developed microservices responsible for different operations such as recording transactions, pricing and generating reports that wer...
• Developed microservices responsible for different operations such as recording transactions, pricing and generating reports that were able to ensure millions of transactions per day using distributed computing.
• Acted as Team Leader during a six months project to create a new product that consisted of several microservices. Dealt with stakeholders and analysts to define the project scope and its deadlines, created and prioritized backlog tasks, responsible for conducting team dailies and developing the new system alongside other team members.
• Worked on replacing a Python API for a Clojure API that was able to handle five times more requests per second.
• Instructed and did the technical onboarding of new employees introducing the system architecture, functional programming, test driven development and event sourcing.