Senior Data Software Engineer with Databricks
We are seeking a Senior Data Software Engineer with Databricks to lead the design, development, and maintenance of scalable data integration pipelines across cloud and enterprise systems. The role focuses on building a Databricks Lakehouse on Azure, driving data ingestion, transformation, orchestration, and automation initiatives across the broader Azure data ecosystem.
Responsibilities
- Lead the design and development of a Databricks Lakehouse on Azure
- Research source systems, then analyze and document their data structures, data flows and business entities
- Build batch and near-real-time ingestion pipelines with Databricks Auto Loader and Azure Event Hubs
- Develop dbt transformations following the medallion architecture
- Design and maintain an operational data store (ODS) on Databricks Lakebase
- Implement integrations with Dataverse and the Power Platform
- Define data engineering standards, code reviews and best practices
- Ensure data quality, testing and documentation across the platform
Requirements
- 5+ years of data engineering experience, including a technical lead role
- Expertise in Databricks, Delta Lake and Apache Spark (PySpark)
- Skills in dbt and Databricks ingestion tools
- Proficiency in Unity Catalog
- Advanced SQL and strong Python skills
- Solid data modeling skills, including dimensional modeling
- Proven ability to analyze unfamiliar source systems and produce clear data documentation
- Familiarity with the Azure data ecosystem: ADLS Gen2, Key Vault and Entra ID
- Experience with Git and CI/CD for data pipelines
- Flexibility to use AI tools
- English proficiency at B2 level or higher
Nice to have
- Background in Databricks Lakebase or PostgreSQL
- Understanding of data governance, Lakehouse and ODS concepts
- Experience with Dataverse, Power Apps or Power Automate integration
- Familiarity with Terraform or Databricks Asset Bundles
- Databricks certifications
- AI certifications
