Lead Databricks Engineer

We’re looking for a Lead Databricks Engineer to join our team in London, UK in a hybrid working mode. This role involves designing and leading enterprise-scale Azure and Databricks data platforms to enable advanced analytics and real-time processing capabilities. You will be responsible for platform architecture, governance, cost optimization and ensuring delivery of high-quality solutions that meet critical business needs in a complex, data-driven environment.

Responsibilities

• Lead architecture design and implementation of scalable solutions on Azure Databricks
• Define and enforce platform standards, ensuring compliance with data governance and security policies
• Build and optimize Databricks streaming workloads, including Structured Streaming and Delta Live Tables
• Collaborate with cross-functional teams to translate requirements into robust technical solutions
• Drive platform performance tuning, cost optimization and monitoring strategies for cloud workloads
• Provide leadership across the full engineering lifecycle, including solution delivery and production support
• Mentor and guide developers, establishing best practices for data engineering on Azure and Databricks
• Participate in technical decision-making and review designs for scalability and maintainability
• Implement observability for critical pipelines and maintain quality assurance standards
• Engage directly with stakeholders to ensure alignment on strategic platform initiatives

Requirements

• Minimum 8+ years of experience in data engineering, including 3+ years in a tech lead capacity
• Strong hands-on expertise in Databricks for enterprise-scale workloads
• Proficiency in PySpark, Spark Structured Streaming, and Delta Lake
• Solid programming skills in Python and demonstrated SQL performance optimization experience
• Deep understanding of Azure Data Platform services and cloud-native engineering patterns
• Proven experience implementing data governance, quality management and observability frameworks
• Ability to manage complex data pipelines for batch and streaming use cases in mission-critical environments
• Strong communication and leadership skills to work with distributed teams and senior stakeholders

Nice to have

• Knowledge of Delta Live Tables (DLT) and advanced workflows in Databricks
• Familiarity with CI/CD for data engineering using GitHub or Azure DevOps
• Experience designing infrastructure-as-code solutions for Azure environments
• Understanding of data mesh or lakehouse principles for large-scale architectures
• Background in financial trading or capital markets data domains