Lead Data Engineer (PySpark)
We are seeking a Lead Data Engineer with deep expertise in Microsoft Azure Fabric–based data platforms and AI-enabled data engineering.
This role blends hands-on technical leadership with architectural and team mentoring responsibilities, focusing on modern data engineering, big data processing, and AI-driven workflows in complex enterprise environments.
Responsibilities
- Lead the design, development and optimization of scalable data engineering solutions using Azure Fabric and cloud-native technologies
- Own end-to-end data pipelines including ingestion, transformation, storage and analytics
- Architect and implement solutions leveraging OneLake (Delta / OpenLake) and Fabric experiences
- Develop and optimize PySpark, SparkSQL and Python-based data processing pipelines
- Work with Cosmos DB (NoSQL API) and other Cosmos DB variants to support high-performance data access patterns
- Implement and maintain CI/CD pipelines and promote DevOps best practices
- Collaborate with data scientists, AI engineers and product stakeholders to enable AI-driven analytics and insights
- Mentor and guide junior engineers, setting coding standards and best practices
- Ensure data quality, security, governance and performance across platforms
- Contribute to technical decision-making and solution architecture discussions
Requirements
- 9+ years of experience in data engineering with AI exposure
- Expertise in Azure Fabric and end-to-end Fabric experience
- Knowledge of OneLake (Delta / OpenLake)
- Advanced skills in Python, PySpark and SparkSQL
- Proficiency in Cosmos DB (NoSQL API) and other Cosmos DB variants
- Understanding of DF Gen2 and M-code
- Capability to implement CI/CD pipelines using Azure DevOps or equivalent tools
- Experience with generic Azure services and Power BI integration, semantic models and performance considerations
- Background in Agile or Scrum environments
- Strong ownership mindset and ability to lead by example
- Excellent communication skills for technical and non-technical audiences
- Good understanding of the financial domain
Nice to have
- Experience with code generation, including non-AI and AI-assisted approaches
- Azure AI Foundry experience
- Data Science fundamentals and collaboration with DS teams
- Strong background in Big Data architectures and Spark ecosystems
- Familiarity with financial instruments and financial services data
- Hands-on experience with industry-standard LLMs (including GPT, Claude or similar)
- Exposure to AI-enabled data platforms and intelligent analytics use cases
