Senior Data Integration Engineer (SQL, Azure, Snowflake)
We are seeking a motivated Senior Data Integration Engineer to join our team and play a key role in building, optimizing, and migrating data pipelines across diverse systems. This role focuses on utilizing SQL, Azure, PySpark, HDFS, and Snowflake to improve data workflows, ensure seamless integrations, and deliver solutions to complex data challenges.
Responsibilities
- Design, develop, and optimize SQL queries and stored procedures for efficient data processing and integration
- Lead data migration projects from on-premises systems or legacy environments to Azure Synapse Analytics
- Work with advanced Snowflake concepts such as window functions, common table expressions (CTEs), and JavaScript-based stored procedures
- Implement data pipelines using PySpark and manage file processing in Hadoop HDFS environments
- Build and troubleshoot Azure Data Factory (ADF) pipelines to integrate data from ADLS and SFTP into target systems
- Collaborate with cross-functional teams to implement and automate end-to-end data workflows
- Monitor and maintain the performance of data pipelines, ensuring scalability and reliability
- Develop and maintain technical documentation for data architecture, transformations, and pipeline processes
- Provide technical leadership and mentoring to junior engineers and team members as needed
Requirements
- 3+ years of experience working with data integration, migration, and pipeline optimization across SQL and cloud platforms
- Expertise in Snowflake, including Snowflake Analytics
- Advanced SQL skills with knowledge of window functions, CTEs, and JavaScript-based stored procedures
- Background in PySpark and Hadoop HDFS, including pipeline migration from Cloudera Data Lake to Azure Synapse Analytics
- Proficiency in creating ADF pipelines for data transfer involving sources such as ADLS and SFTP
- Competency in Python, PySpark, and SQL for data engineering tasks and automation
- Understanding of Azure Cloud services and expertise in tools like Azure Synapse Analytics and Power Automate
- English level of minimum B2 (Upper-Intermediate) for effective communication
Nice to have
- Familiarity with legacy systems and experience handling data integration from on-prem environments to cloud platforms
- Showcase of performance tuning techniques for Snowflake stored procedures and SQL pipelines
- Qualifications in additional Azure services or certifications
