Senior Data Engineer

FreightVerify is a supply chain technology company that provides real-time transportation visibility and business intelligence for global enterprise clients. Utilizing the latest IoT, big data, and AI technologies, FreightVerify provides a neutral platform that simplifies complex global supply chains through a common design language to leverage new and emerging tracking technologies.

As a Senior Data Engineer, you will develop, construct, test, and maintain architectures, such as databases and large-scale processing systems for AI and machine learning. Besides having a deep passion for data and data engineering, you will need to have extensive experience with software development, particularly in Python and R. The position has a distinct emphasis on data architecture and data lakes, so the ideal candidate will have experience designing and implementing Hadoop-based system.

An ability to clean, massage, and organize big data is also a necessary part of your background. Working with a team of AI engineers and data scientists, you will work with raw data from multiple sources to be cleaned and integrated, with an eye towards data reliability, efficiency, and quality. You will need to employ various languages and tools to connect systems and acquire new data from other sources.

You will analyze complex problems and opportunities, propose solutions and help lead teams to implement algorithmic enhancements across our platform. In this role, you will support and contribute to the development of a data architecture for AI and machine learning.

Responsibilities

• Develop, construct, test, and maintain large-scale data architectures (e.g., data lakes and large- scale processing systems).
• Ensure our big data architecture will support the requirements of the FreightVerify platform.
• Discover opportunities for data acquisition and new data source integrations.
• Develop dataset processing for data modeling, mining, and production, including serverless solutions.
• Experience with managed cloud platforms such as Azure (preferred), AWS, Google Cloud.
• Employ a variety of languages and tools (Python, R, Scala, etc.) to connect systems.
• Work with tools such as Hadoop, Hive, MySQL, Redis, Riak, MongoDB, neo4j, and Sqoop, to architect and maintain complex data storage solutions.
• Recommend ways to improve data reliability, efficiency, and quality.
• Communicate with stakeholders (clients, product managers, engineers, and data scientists), get feedback, and understand their needs.
• Keep an eye on the results, monitoring them using dashboards for model performance, and checking the health of the services with the support of monitoring systems such as Datadog and New Relic.

Qualifications

• 5+ years of data engineering/software development experience. An MS in analytics/data architecture-related discipline (Computer Science, Engineering, Math, Physics, or other quantitative sciences).
• Strong leadership skills and experience building and leading a group of talented data and software engineers.
• Hands-on experience designing and building data lakes and data warehousing solutions. Experience with Microsoft Azure is a plus.
• Experience in managing data science projects. Ability to get an idea from inception to implementation, including knowledge of validation and A/B testing techniques. Software development skills in Python for Data Science.
• Knowledge of analytics techniques (machine learning/time series analysis/optimization, etc.) commensurate with education/experience level.

We are an equal opportunity employer, hiring and developing individuals from diverse backgrounds and experiences to add to our collaborative culture. We do not discriminate in our recruiting, hiring, and promoting processes. Candidates and employees are treated with respect regardless of race, religion, sex, age, sexual orientation, gender identity and expression, national origin, veteran status, or disability.