Scala Developer Lead

Responsibilities

• Design & develop Scala/Spark processes for data discovery

• Produce unit tests for Spark transformations and helper methods

• Write Scaladoc-style documentation with all code

• Design data processing pipelines

Skills Required

• Scala (with a focus on the functional programming paradigm)

• Apache Spark 2.x

• Apache Spark RDD API

• Apache Spark SQL DataFrame API

• Apache Spark Streaming API

• Containerization experience (docker & Kubernetes)

• Spark query tuning and performance optimization

• SQL database integration (Microsoft, Oracle, Postgres, and/or MySQL, etc)

• Experience working with HDFS, S3, Cassandra, and/or DynamoDB

• Experience with document processing under Spark Streaming

• Experience with Kafka & Zookeeper

• Understanding of distributed systems