Data DevOps Engineer
We are seeking a talented and motivated Data DevOps Engineer to support the operationalization and maintenance of an Enterprise Data Platform (EDP) Foundation, ensuring a secure and efficient environment for scalable data-driven operations.
Responsibilities
- Install and configure platform components to enable integration within the Enterprise Data Platform stack
- Set up RBAC to enforce granular permissions and support security best practices
- Design and maintain CI/CD pipelines using tools like GitLab CI, integrating infrastructure provisioning with technologies such as Terraform
- Establish Logging and Monitoring capabilities with the LGTM stack for improved observability and performance tracking
- Build and deploy a centralized Single Management Console for streamlined platform management
- Enable multi-tenancy capabilities to provide independent and secure environments for different users, teams, and locations
- Automate data ingestion, transformation, and querying workflow pipelines
- Optimize platform infrastructure using Kubernetes and Red Hat OS to achieve high availability and scalability
- Utilize HashiCorp Vault and Open Policy Agent to enforce security policies and maintain compliance standards
- Work with engineers and platform teams to enhance release processes and support continuous delivery
- Monitor, troubleshoot, and resolve issues to ensure operational reliability and system performance
- Collaborate with customer technical teams to achieve project goals and milestones
Requirements
- 3+ years of experience with Kubernetes for container orchestration and Red Hat OS in enterprise environments
- Background in Apache Kafka, MinIO, Apache Iceberg, and Apache Spark, including Spark Streaming
- Proficiency in managing distributed SQL query engines like Trino and databases such as PostgreSQL
- Skills in Terraform for infrastructure provisioning and GitLab CI or similar tools for automated deployment pipelines
- Knowledge of HashiCorp Vault and Open Policy Agent for secure access control and policy enforcement
- Expertise in Logging and Monitoring tools, particularly the LGTM stack (Loki, Grafana, Tempo, Mimir)
Technologies
- Data Platform Components:
- Apache Kafka
- Apache Spark (including Spark Streaming)
- MinIO (Object Storage compatible with S3)
- Apache Iceberg (Table format for analytical datasets)
- PostgreSQL
- Trino (Distributed SQL query engine for big data)
- Infrastructure & Security:
- Red Hat OS
- Kubernetes for container orchestration
- HashiCorp Vault for secrets and credential management
- Open Policy Agent (OPA) for policy enforcement
- Logging and Monitoring:
- LGTM stack (Loki, Grafana, Tempo, Mimir)
