Senior Data Platform Engineer (Databricks)
We are looking for a seasoned Senior Data Platform Engineer to join EPAM, a company at the forefront of shaping digital solutions for Fortune 1000 businesses.
As a critical part of our team, the ideal candidate will bring deep expertise in Databricks and experience across multi-cloud environments. This role will focus on building, optimizing, and maintaining innovative, high-performance data platforms that enable powerful data insights and drive meaningful business transformations.
This position offers a hybrid work setup with flexibility to work from Malaga or Madrid, with occasional office visits required.
Responsibilities
- Architect and deploy robust data platforms using Databricks, focusing on optimal performance and security
- Create solutions that are cloud-agnostic across AWS, Azure, and GCP to ensure system flexibility and resilience
- Design and implement comprehensive data pipelines involving data lakes, warehouses, and streaming technologies
- Utilize Databricks SQL, Delta Lake, MLflow, and Spark for data interaction and performance enhancements
- Collaborate with various teams to implement and maintain workflows based on Databricks best practices
- Develop CI/CD pipelines tailored for data platform deployment and testing
- Set up and manage frameworks for monitoring, logging, and alerting to ensure infrastructure health
- Optimize compute and storage resources to balance cost-efficiency and performance
- Troubleshoot Databricks and Spark performance issues
- Mentor team members on effective cluster management and resource allocation in Databricks environments
- Maintain compliance and security standards throughout platform operations
- Drive adoption of advanced Databricks capabilities like Photon and Graviton instances
- Regularly refine architectures to align with evolving business and technology requirements
Requirements
- Extensive experience in Databricks, Apache Spark, and distributed data processing systems
- Strong programming skills in Python, Scala, SQL
- Proficiency in data engineering services on AWS (S3, IAM, Lambda), Azure, GCP
- Expertise in data architecture and ETL workflows, focusing on data lakes and lakehouses
- Hands-on experience with Terraform, CloudFormation, and CI/CD tools
- Familiarity with monitoring tools and observability frameworks for large-scale data environments
- Solid communication skills in Spanish (at least C1)
Nice to have
- Certifications in Databricks, AWS, Azure, GCP
- Knowledge of Kubernetes and containerized deployments for data pipelines
- Experience with real-time data streaming frameworks and governance tools
