Data Engineer / Senior Data Engineer with Azure, Databricks and Microsoft Fabric
About the Position
We are looking for a Senior Data Engineer with strong experience in Azure, Databricks, and Microsoft Fabric to help design and build a modern cloud data platform. You will contribute to the ingestion, transformation, and curation layers of the platform while collaborating with technical and business stakeholders. In this role, you will also support engineering standards, platform design decisions, and the delivery of scalable data solutions.
About the Project
This project focuses on building a modern enterprise data and analytics platform on Microsoft Azure as part of a large scale modernization program. A legacy policy administration system is being replaced with a SaaS solution, while reporting and analytics capabilities are being migrated from a traditional SQL Server environment to Databricks and Microsoft Fabric. The platform will support governed data products, advanced analytics, and AI assisted engineering practices.
Responsibilities
- Design and develop data ingestion pipelines from business systems into a governed cloud data platform.
- Build scalable data engineering solutions using Azure Databricks and Microsoft Fabric.
- Implement medallion architecture patterns and reusable engineering frameworks.
- Develop curated data products and analytical datasets for business reporting and decision making.
- Apply AI assisted development practices while ensuring quality, governance, and auditability.
- Contribute to data modelling, semantic models, metadata management, and analytical solutions.
- Support the migration of reporting and analytics workloads from legacy platforms.
- Collaborate with stakeholders to translate business requirements into technical solutions.
- Provide design guidance, establish engineering standards, and participate in code reviews.
Requirements
- 6+ years of experience in Data Engineering.
- Hands on experience with Azure Data Factory, Azure Databricks, and Azure Data Lake Storage Gen2.
- Strong programming experience with Python, SQL, and PySpark.
- Experience implementing Delta Lake and production lakehouse or medallion architectures.
- Experience with Git and Azure DevOps, including CI/CD pipelines for data solutions.
- Experience working with SQL Server environments.
- Practical experience using AI coding assistants within software delivery projects.
- Ability to communicate effectively in English with technical and business stakeholders.
- Experience taking ownership of architecture or platform design decisions.
- Experience reviewing code and establishing engineering standards.
- Experience gathering requirements and collaborating directly with business stakeholders.
Nice to Have
- Experience implementing Microsoft Fabric in production environments.
- Microsoft certifications including DP 203, DP 600, DP 700, AZ 900, or DP 900.
- Databricks Data Engineer Associate or Databricks Data Engineer Professional certification.
- Experience with dimensional modelling, including star schema, snowflake schema, or Data Vault.
- Experience with semantic modelling in Power BI or Microsoft Fabric.
- Knowledge of Microsoft Purview, data lineage, and metadata management.
- Experience working in regulated financial services environments.
Technologies
Azure Data Factory, Azure Databricks, Microsoft Fabric, Azure Data Lake Storage Gen2, Delta Lake, Apache Spark, PySpark, Python, SQL Server, Azure DevOps, Git, Power BI, Microsoft Purview, GitHub Copilot
