Data Engineer
(2+ years exp)$125k – $175k
Published: 1 month ago
Viaduct
AI for connected vehicles
Job Location
Job Type
Full TimeVisa Sponsorship
Not AvailableHires remotely
Everywhere
Relocation
AllowedSkills
Python
SQL
Distributed Systems
Apache Spark
Apache Airflow
The Role
Who You Are
You are a thoughtful engineer. You understand the complexities of distributed systems and how to triage and solve issues that arise with them. Scalability is top of mind when designing any system or writing code. You believe building a better ETL system requires close collaboration with the machine learning and data science teams. You avoid reinventing the wheel unless necessary and are excited by opportunities to contribute to the open-source community.
Day 5
- Learn about Viaduct’s history and mission
- Get to know every team member
- Set up your development environment
- Understand Viaduct’s ETL pipelines and run your first DAGs
- Deep dive into the nuances of vehicle data
- Attend our weekly ML lunch
Day 30
- Take ownership of ETL pipelines
- Identify scalability bottlenecks in the existing ETL pipelines
- Be familiar with the day-to-day work of machine learning engineers and data scientists
- Learn the architecture of data engineering systems and services
Day 90
- Be the ETL pipeline expert at Viaduct
- Improve overall data quality and discoverability
- Confident in the scalability of Viaduct’s ETL pipelines
- Present your work at our weekly ML lunch
- Comfortable contributing to our engineering infrastructure and systems
Expected Skills
- 2+ years working with large-scale data processing tools (Spark, Hadoop, Airflow, etc)
- Expertise in Python, Scala, or Go
- Experience managing Spark clusters and tuning Spark jobs
- Active user of and/or contributor to open-source projects
- Exceptional Skills
- Familiar with Terraform, Docker, and Kubernetes
- Familiar with managing data engineering infrastructure (Airflow, Kubernetes, etc)
Why Viaduct
- Contribute to the open-source ecosystem
- Work with established experts in deep learning, time-series analytics, and convex optimization
- Endless opportunities for technical learning and personal growth
- Full health, vision, and dental benefits
Similar Jobs
Node.io
Discover your next opportunity
Home Delivery Service (HDS Global)
Personalized eCommerce, featuring touchless fulfillment – starting with fresh groceries
GVOS
An Edge Cloud for Autonomous Driving
Marco
Trade Finance Platform for SMEs
Forward
Forward combine hardware, software and doctors to make quality healthcare available to all
Above Data
Platform to accelerate business decisions from transaction data
PatternAI
DSaaS - Data Science as a Service
CipherTrace
We are growing the crypto economy by making virtual assets safe and trusted
AskWhai
Helping humans navigate a fast-changing world and reach their maximum potential