We are seeking a skilled Senior/Lead Data Quality Engineer to join our team. The successful candidate will play a crucial role in ensuring the integrity, reliability, and quality of our data engineering pipelines and platforms.
Responsibilities
- Design and develop a robust automated testing framework and strategy for data engineering pipelines, with a focus on integration with Databricks, PySpark, Scala, Spark SQL, and other relevant technologies
- Implement automated tests to ensure data integrity, quality, and reliability across all data pipelines and platforms
- Integrate automated testing within Azure DevOps CI/CD pipelines to ensure seamless testing, build, and deployment processes
- Conduct and execute performance tests to assess the efficiency and scalability of data processing systems
- Work closely with data engineers and other stakeholders to understand requirements and ensure testing strategies are aligned with business objectives
- Establish monitoring for automated tests, providing regular reports on test coverage, defects, and data quality issues
Requirements
- Minimum of 3 years of experience in Data Quality Engineering or a related field
- Solid understanding of automated testing frameworks and data engineering principles
- Proficiency with Databricks, PySpark, Scala, Spark SQL, Terraform, and Azure Event Hubs
- Experience in integrating testing frameworks with Azure DevOps and managing source control with Git
- Knowledge of pipeline configuration and deployments utilizing Terraform
- Familiarity with Jira
- Strong analytical abilities with a proven track record in troubleshooting and resolving complex data and testing challenges
- Fluency in English with a B2+ level
Nice to have
- Experience with Power BI, Azure Data Lake, Spark Streaming, and other data processing and visualization tools
- Prior experience in a data-intensive industry or with large-scale data environments
- Proficiency in implementing solutions through Infrastructure as Code (IaC)