Lead Apache Airflow Engineer

We are seeking a Lead Apache Airflow Engineer to spearhead the development and enhancement of our workflow orchestration platform. In this role, you will drive the evolution of Airflow components, integrate new versions with Cloud Composer and lead a talented team of engineers in building robust data pipelines.

Please note that working from the office in Gdansk or Wroclaw 3 days per week is required.

Responsibilities

• Develop and maintain Airflow Operators, Hooks and Sensors
• Integrate new Airflow versions with Cloud Composer
• Fix bugs in Airflow and implement improvements
• Implement tests for Airflow, including Unit Testing and Solution Testing
• Create internal documentation for work artifacts
• Lead and mentor a team of Airflow engineers
• Collaborate with Open Source communities to contribute improvements

Requirements

• 5+ years of experience in Python
• 1+ years of relevant leadership experience
• Programming experience with Cloud APIs (REST/RPC)
• Basic knowledge of DevOps/SysAdmin
• Basic understanding of Linux and Kubernetes, including how to build container images
• Familiarity with Cloud platforms
• Expertise in Airflow or similar experience in building workflows (e.g. based on Luigi, Oozie, Argo, Azkaban, Apache HiFi)
• Experience working with Open Source communities
• Communication skills in English, spoken and written (B2+)

Nice to have

• Understanding of Cloud Architecture, including scalability and IaaS principles