How and why I built AWS Hybrid Data Ecosystem

Published May 16, 2018Last updated May 25, 2018

About me

I am a Data Engineer with a passion for analytics and building advanced streaming analytics applications

The problem I wanted to solve

Allow data to be ingested from on prem to cloud, then allow third parties to consume from the cloud

What is AWS Hybrid Data Ecosystem?

End to end solution to allow ingestion and consumption

Tech stack

Spring Boot, AWS S3, Kafka, AWS Athena

The process of building AWS Hybrid Data Ecosystem

Visulaized an idea, leveraged existing components, implemented it at scale

Challenges I faced

Security and Authentication

Key learnings

Best practices of when to be on prem and when to be in the cloud

Tips and advice

Utilize the power of AWS SDK

Final thoughts and next steps

Cloud storage can solve 90% of problems with developing a datalake

Discover and read more posts from William Griffith
get started