Lead Data Engineer (Databricks & Spark)
We are building a next-generation analytics platform and need a Lead Data Engineer to shape scalable ingestion, transformation, and governed data products on Databricks. You will combine hands-on delivery with technical leadership and architecture to enable self-service analytics and AI-assisted engineering.
Responsibilities
- Design scalable data ingestion and transformation frameworks on Databricks
- Define and enforce Databricks best practices for medallion layers and operational excellence
- Build batch and streaming pipelines using Apache Spark and PySpark
- Deliver end-to-end data products from ingestion through governed consumption layers
- Develop trusted analytical datasets and semantic models for reporting and self-service analytics
- Implement data quality, lineage, monitoring, and observability standards
- Lead proof-of-concepts and evaluate emerging capabilities in the Databricks ecosystem
- Mentor engineers and drive reusable frameworks adopted across teams
Requirements
- 5+ years of data engineering experience with production-grade platforms
- Strong Databricks experience in enterprise production environments
- Lead-level technical leadership skills to drive standards and mentor engineers
- End-to-end delivery experience across ingestion, transformation, semantic layers, and reporting
- Advanced Apache Spark and PySpark expertise for scalable pipeline development
- Advanced SQL skills for transformation, modeling, and performance tuning
- Upper-Intermediate English proficiency (B2) for leading technical discussions
Nice to have
- Data architecture experience for enterprise lakehouse patterns
- Data solution architecture skills to define scalable platform designs
- Scala proficiency for Spark development and optimization
