Lead Data Software Engineer (Databricks)
We are searching for a Lead Data Software Engineer to join our team and oversee the development of our cutting-edge supply-chain data analytics platform.
This platform delivers a comprehensive view of suppliers, products, material categories, shipments, and compliance activities while employing artificial intelligence to generate insights and recommendations tailored to our clients’ needs.
If this role excites you – apply!
Responsibilities
- Design and develop data architectures and ETL pipelines utilizing Databricks and external orchestrators such as Airflow
- Collaborate with machine learning teams to incorporate AI-driven insights into the platform
- Develop system-level improvements and components for data engineering workflows
- Optimize the performance and scalability of data-intensive applications
- Adhere to software engineering standards, including containerization, unit testing, linting, and code style reviews
- Maintain and improve data workflow, Delta Lake, and Delta Live Tables within Databricks
- Provide technical mentorship and leadership to the data engineering team
- Engage with clients and stakeholders to understand their requirements and offer customized solutions
- Tackle challenges independently and ensure successful project delivery
Requirements
- Experience leading data engineering efforts on projects utilizing Databricks
- Solid understanding of data architectures and skills in data modeling
- Background in designing and building ETL pipelines in Databricks with external orchestrators such as Airflow
- Hands-on expertise in Databricks (Delta Lake, workflows, Delta Live Tables, deployment, and versioning)
- Proficiency in Python and cloud-native technologies
- Competency in Spark/PySpark
- Engineering experience with either AWS or Azure
- Knowledge of big data and techniques for optimizing data-intensive applications
- Strong communication skills with a proactive attitude and client-facing experience
- Ability to navigate ambiguity and complete projects independently
- Comfort working in a dynamic and transparent startup environment
- Fluent English communication skills at a C1 level
Nice to have
- Proficiency in configuring and managing CI/CD pipelines in Azure DevOps
- Familiarity with Data Observability and Data Quality Monitoring practices
- Capability to integrate data quality checks into data pipelines
