Data Pipeline Development Engineer
Project description
We are seeking experienced Software Engineering Analysts / Data Workflow Engineers to design, implement, and operate scalable data orchestration workflows using Apache Airflow. You will work as part of an agile engineering team, contributing to the full lifecycle of workflow development—from concept and design to implementation, operation, and continuous improvement. This role requires strong technical expertise in Python, workflow orchestration, and system integration, along with the ability to collaborate closely with cross-functional software teams.
Responsibilities
- "
- Analyze existing process flows and architecture diagrams to design scalable Airflow workflows
- Design end-to-end orchestration solutions, including integrations with databases, APIs, and external systems
- Implement and configure Apache Airflow DAGs using Python
- Migrate existing Jenkins pipelines and workflows to Apache Airflow
- Operate, monitor, and maintain the Airflow scheduler and workflows in production
- Troubleshoot workflow failures and performance issues
- Ensure solutions meet robustness, reliability, and performance expectations
- Actively participate in agile ceremonies, governance meetings, weekly status updates, and daily stand-ups during active delivery phases
- Collaborate closely with internal development teams and stakeholders"
SKILLS
Must have
- Minimum 3 years of hands-on experience with Apache Airflow (ETL/ELT workflow development and orchestration)
- Strong proficiency in Python
- Experience integrating multiple data sources (SQL databases, REST APIs, cloud services)
- Experience working in agile development environments
- Familiarity with CI/CD pipelines and DevOps practices
- Strong ability to understand complex technical workflows and suggest improvements
- Excellent written and spoken English, especially for technical documentation
Nice to have
Meritorious / Nice to Have: Experience with cloud platforms (Azure, AWS, GCP); experience operating Airflow in enterprise or large-scale environments; prior experience migrating workflows from Jenkins or similar orchestration tools.
