Senior Data Engineer with Java, Apache Spark

We are seeking a Senior Data Engineer to join a technology-driven organization operating in the financial data domain, building a production-grade analytics and data ingestion platform processing large-scale financial datasets. This role focuses on designing, building and operating large-scale data ingestion and analytics solutions for batch and real-time processing of financial data, supporting critical downstream analytics and reporting use cases.

Responsibilities

• Design, develop and maintain Apache Spark batch pipelines and Kafka streaming solutions
• Build serverless microservices using AWS Lambda and Azure Functions
• Develop data ingestion connectors for databases, file systems, message queues and financial data feeds
• Write production-grade code in Java, Scala and Python
• Manage infrastructure using Terraform across multiple environments
• Containerize and deploy applications using Docker and Kubernetes (EKS/AKS)
• Implement and maintain CI/CD pipelines in GitLab CI
• Write and maintain unit and integration tests with high code coverage
• Monitor platform health and troubleshoot production issues in distributed systems

Requirements

• 5+ years of experience in data engineering or backend platform development
• Proficiency in Java and/or Scala as primary languages with Python as secondary
• Expertise in Apache Spark including Spark SQL and Spark Streaming
• Skills in Apache Kafka and event-driven architectures
• Competency in SQL and data modeling
• Familiarity with AWS cloud services and serverless computing

Nice to have

• Knowledge of Terraform for Infrastructure as Code
• Skills in Docker and Kubernetes (EKS/AKS)
• Familiarity with CI/CD pipelines such as GitLab CI, Jenkins or equivalent
• Background in cloud data warehouses such as Amazon Redshift
• Understanding of Microsoft Azure services and Azure-native tooling
• Knowledge of Iceberg, Hudi or Delta Lake