Lead Data Engineer
We're looking for a Lead Data Engineer to construct an MLOps platform that supports automated model delivery, deployment, and large-scale feature engineering. In this role, you'll advance the featureOps stack by leveraging Azure Databricks, Azure Data Factory, and Azure DevOps to implement CI/CD pipelines feeding into the Databricks Feature Store for Model Lab applications.
Responsibilities
- Transition existing Spark feature engineering pipelines over to Azure
- Work alongside the team to design and develop a forecasting model
- Support the deployment of the machine learning model into a production environment
- Contribute to designing and building the CI/CD pipeline that supplies the Databricks Feature Store
- Work in close coordination with Data Scientists and engineers to align feature engineering with ML model requirements
- Ensure data pipelines remain reliable and scalable across the entire platform
- Troubleshoot and enhance current Spark-based workflows during the migration process
Requirements
- A minimum of 5 years of practical PySpark experience
- Strong foundational knowledge in core data science principles
- Demonstrated success in building feature engineering and machine learning pipelines
- Hands-on background in moving data pipelines into cloud-based environments
- B2 (Upper-Intermediate) English proficiency or above
Nice to have
- Familiarity with MLflow and Docker
- Working knowledge of Azure Kubernetes Services, Azure Data Factory, and Azure Databricks
- Understanding of best practices for deploying ML models
