Senior Data Software Engineer
We are searching for a Senior Data Software Engineer to guide the development of our forward-thinking supply-chain data analytics platform.
This platform offers a comprehensive perspective of suppliers, products, material categories, shipments, and compliance activities, leveraging artificial intelligence to provide actionable insights and custom recommendations for our clients.
If this role matches your career aspirations – apply today!
Responsibilities
- Design and develop data architectures and ETL pipelines using Databricks and external orchestrators like Airflow
- Collaborate with machine learning teams to incorporate AI-driven insights into the platform
- Implement system-level enhancements and components for efficient data engineering
- Optimize data-focused platforms to enhance performance and scalability
- Ensure compliance with software engineering best practices, including containerization, unit testing, linting, and code style adherence
- Oversee and improve data workflows, Delta Lake, and Delta Live Tables within Databricks
- Provide technical mentorship and support to data engineers on the team
- Engage effectively with clients and stakeholders to align solutions with their needs
- Navigate uncertainty and independently complete project deliverables
Requirements
- Background in leading data engineering projects with the Databricks platform
- Knowledge of data architectures and data modeling
- Skills in designing and building ETL pipelines utilizing Databricks and external orchestrators (e.g., Airflow)
- Proficiency in using Databricks (Delta Lake, workflows, Delta Live Tables, deployment, and versioning)
- Expertise in Python with a focus on cloud-native technologies
- Proficiency in Spark/PySpark
- Background in engineering with either AWS or Azure
- Skills in handling big data and optimizing performance in data-intensive environments
- Strong initiative combined with experience interacting directly with clients
- Adaptability to manage ambiguity and independently drive outcomes
- Competency in operating within a fast-paced startup environment with transparency
- Fluency in English at a B2 level
Nice to have
- Familiarity with setting up or maintaining CI/CD pipelines on Azure DevOps
- Understanding of Data Observability and approaches to Data Quality Monitoring
- Background in integrating data quality checks into data pipelines
