Senior Data Engineer (Databricks)

EPAM·Malaysia·Офис·5д. назад

We are looking for an experienced Senior Data Integration Engineer specializing in Databricks to join our team. As a key member of the team, you will lead the design of enterprise digital products and platforms that deliver value to users and achieve business goals. You will collaborate with various stakeholders and contribute to creating engaging and delightful solutions by leveraging your design skills and methods. Come and join us to promote the role of design and achieve positive design outcomes.

Responsibilities

Develop, maintain and support ETL/ELT solutions
Implement and manage Databricks on Microsoft Azure or AWS Cloud platforms
Use SQL, PySpark or Scala to transform and analyze large datasets
Own end-to-end delivery of data ingestion + transformation pipelines on Azure Databricks
Works independently with minimal supervision
Mentoring junior engineers
Work collaboratively with the team on Microsoft Power BI
Ensure the adherence to best practices in development, testing and deployment
Create technical documentation related to the project

Requirements

At least 5 years of experience and strong experience in Data Integration, proficient in Databricks and ETL/ELT Solutions
Strong SQL/Python, data modeling fundamentals (dimensional / Lakehouse modeling), incremental processing, data transformation and analysis
Experience with Delta Lake concepts (tables, partitions, schema evolution)
Familiarity with cloud platforms (Azure, AWS)
Have hands-on experience developing data pipelines on Databricks, integrating data from multiple sources, and working closely with analytics, data science and business teams to deliver reliable, high-quality data products
Knowledge of data ingestion techniques and data integration best practices
Orchestration experience (Databricks Workflows; ADF/Airflow is a plus)
Excellent verbal and written communication skills in English
No visa sponsorship available