Data DevOps Engineer

We are seeking a talented and motivated Data DevOps Engineer to support the operationalization and maintenance of an Enterprise Data Platform (EDP) Foundation, ensuring a secure and efficient environment for scalable data-driven operations.

Responsibilities

• Install and configure platform components to enable integration within the Enterprise Data Platform stack
• Set up RBAC to enforce granular permissions and support security best practices
• Design and maintain CI/CD pipelines using tools like GitLab CI, integrating infrastructure provisioning with technologies such as Terraform
• Establish Logging and Monitoring capabilities with the LGTM stack for improved observability and performance tracking
• Build and deploy a centralized Single Management Console for streamlined platform management
• Enable multi-tenancy capabilities to provide independent and secure environments for different users, teams, and locations
• Automate data ingestion, transformation, and querying workflow pipelines
• Optimize platform infrastructure using Kubernetes and Red Hat OS to achieve high availability and scalability
• Utilize HashiCorp Vault and Open Policy Agent to enforce security policies and maintain compliance standards
• Work with engineers and platform teams to enhance release processes and support continuous delivery
• Monitor, troubleshoot, and resolve issues to ensure operational reliability and system performance
• Collaborate with customer technical teams to achieve project goals and milestones

Requirements

• 3+ years of experience with Kubernetes for container orchestration and Red Hat OS in enterprise environments
• Background in Apache Kafka, MinIO, Apache Iceberg, and Apache Spark, including Spark Streaming
• Proficiency in managing distributed SQL query engines like Trino and databases such as PostgreSQL
• Skills in Terraform for infrastructure provisioning and GitLab CI or similar tools for automated deployment pipelines
• Knowledge of HashiCorp Vault and Open Policy Agent for secure access control and policy enforcement
• Expertise in Logging and Monitoring tools, particularly the LGTM stack (Loki, Grafana, Tempo, Mimir)

Technologies

• Data Platform Components:
• Apache Kafka
• Apache Spark (including Spark Streaming)
• MinIO (Object Storage compatible with S3)
• Apache Iceberg (Table format for analytical datasets)
• PostgreSQL
• Trino (Distributed SQL query engine for big data)
• Infrastructure & Security:
• Red Hat OS
• Kubernetes for container orchestration
• HashiCorp Vault for secrets and credential management
• Open Policy Agent (OPA) for policy enforcement
• Logging and Monitoring:
• LGTM stack (Loki, Grafana, Tempo, Mimir)