Josiah Berkebile

Mostly writing Spark processing pipelines on very large (1/2 petabyte or more) datasets.

I'm playing a lead role in getting AMC Theatres Big Data initiative off the ground. Responsibilities and Accomplishments: - Extended a Spark Sentiment Analyzer written in Scala using Stanford CoreNLP to analyze complex customer feedback. - wrote a custom Flume Source plugin in Java and Scala + Cats for ingesting a vendor's realtime HTTPS event stream - used Scala, Akka, Scalatra, and Cats to develop an HTTP-based Custom Flume Client - Co-Administrator of a CDH5 (Cloudera) cluster - Training for Hadoop software development and Scala programming to peers/engineers - Development process and workflow advisor - Exploratory research and project idea generation - Develop new solutions/Apps leveraging Hadoop technologies including Flume, Spark, Impala, Hive, and HBase - Deploy new Hadoop apps and plugins to a Kerberized CDH 5 cluster - Rig applications to execute through Sysvinit, Upstart, or Systemd - Rig system-initiated applications to auto-authenticate to Kerberos using keytabs - Automation Engineer and advisor - Haskell-style functional programming in Scala using Cats - Imported deeply nested JSON files into Hive and Impala and flattened it out into a traditional SQL table structure. - wrote real-time data ingestion to HDFS apps using Linux Shell scripting, Python, Java, and Scala - Created a Docker CDH 5 development sandbox for prototyping.

Automated full SQL translations of databases with schemas containing thousands of schemas and over 100 thousand queries using Python. This Python translated SQL between multiple data stores such as Teradata, Hive LLAP, and HAWQ/Greenplum. Ran performance profiling, tuning, and analysis on Hive LLAP and HAWQ/HDP/Greenplum databases. Wrote a data comparator using Python and Pandas which provided the differences between data sets and the reason for each difference (precision error vs incorrect value vs NULL value, etc). Wrote a concurrent Python script for automating the transfer of data from Hive into both Netezza and Hbase. Data extraction and data loading were both done using dynamic concurrent processing in Python. PoC'd a Spark SQL on top of Hive LLAP application.