Senior Data Engineer, Databricks/Kafka
We are looking for a Senior Data Engineer to design, implement and optimize scalable data pipelines and infrastructure. You will play a key role in ensuring data availability, reliability and quality for analytics, machine learning and business intelligence applications. Your work will involve collaborating with data scientists, analysts and software engineers to develop robust ETL/ELT workflows, real-time streaming solutions and cloud-native data architectures.
Responsibilities
- Architect, build and optimize scalable data pipelines for batch and real-time data processing
- Develop and implement ETL/ELT workflows, ensuring efficient data ingestion, transformation and storage
- Leverage modeling (Party model, Datavault) methodologies to enable scalable and flexible data modeling
- Ensure data consistency, reliability and governance across data lakes, warehouses and operational data stores
- Optimize performance and cost efficiency of data infrastructure on Azure
- Implement and manage big data processing frameworks, such as Databricks, Kafka
- Enhance data security and compliance, integrating RBAC & ABAC, encryption and regulatory frameworks (GDPR) into data infrastructure
- Develop automation tools for data pipeline orchestration, using Airflow, Azure Data Factory or Prefect
- Monitor, troubleshoot and optimize data pipelines, ensuring minimal downtime and quick recovery from failures
- Collaborate with federated engineering teams to align data architecture with business and engineering goals
- Provide clean, reliable and scalable datasets for advanced analytics and machine learning in partnership with data scientists and business analysts
- Evaluate and adopt emerging technologies, ensuring continuous improvement in data engineering best practices
- Mentor and guide junior engineers, fostering technical excellence and knowledge sharing within the team
Requirements
- 5+ years of experience in data engineering, ETL development or big data technologies
- Expertise in designing and optimizing ETL/ELT workflows, using tools such as dbt, Airflow, Azure Data Factory or Apache NiFi
- Hands-on experience with cloud-native data platforms, including Azure Synapse, Databricks, Snowflake or BigQuery
- Knowledge of data modeling techniques, including Data Vault 2.0, star schema and normalization strategies
- Experience with large-scale distributed computing frameworks (Apache Spark, Hadoop, Kafka, Event Hub)
- Advanced proficiency in SQL and programming languages, such as Python, Scala or Java
- Understanding of Infrastructure as Code (Terraform, Pulumi, ARM templates) for managing cloud-based data infrastructure
- Skills in data security and governance best practices, including RBAC, encryption and data lineage
- Experience working in a federated engineering environment, ensuring seamless integration across teams
- Proficiency in observability and monitoring tools for data pipelines, such as Monte Carlo, DataDog and Great Expectations
- Familiarity with Agile and DevSecOps methodologies, ensuring continuous integration, deployment and monitoring of data solutions
- Bachelor's or Master's degree in Computer Science, Data Engineering, Information Systems or a related field
- Upper-Intermediate English language proficiency (B2+)
