BigData Test Automation Specialist

EPAM·Switzerland·Удалённо, Офис·4д. назад

We are looking for a BigData Test Automation Specialist to join our team in Zurich. This role focuses on designing and implementing a robust, multi-layered testing strategy for a modern big data platform within the banking sector. You will work with cutting-edge technologies including Trino, Kafka, Iceberg and Dataiku to ensure data quality, platform resilience and overall system integrity.

Responsibilities

Design and implement comprehensive test automation strategies for a big data platform
Develop and maintain scalable test frameworks
Implement declarative data quality checks for schemas
Create and manage automated tests for complex SQL transformations
Test and validate Hive metastore schemas and data integrity
Develop tests for streaming data pipelines using Kafka and Testcontainers
Automate Dataiku DSS scenarios and integrate them into CI/CD pipelines
Build and optimize CI/CD infrastructure for efficient big data testing
Perform resilience and chaos testing to ensure platform stability
Use data versioning patterns like Write-Audit-Publish for isolated and safe testing

Requirements

Strong experience with Python and the Pytest framework for test automation
Hands-on experience with big data technologies like Trino, Hive, Kafka or S3
Proficiency with data quality and validation tools
Experience with containerization and mocking tools like Docker, Testcontainers
Proven ability to test streaming applications and event-driven architectures
Knowledge of scripting for cloud object storage interaction
Experience with MLOps platforms, preferably Dataiku
Proven experience building and managing CI/CD pipelines (e.g. GitHub Actions, GitLab CI)
Familiarity with SQL transformation tools like dbt or SQLMesh
Understanding of data versioning concepts
Fluent in English