Senior/Lead Data Engineer
We are looking for a Senior/Lead Data Engineer to build telemetry and cost-attribution data pipelines that map real-time accelerator utilization to workloads and teams.
Responsibilities
- Build BigQuery/Dataflow pipelines ingesting TensorCore utilization, FLOPs, and network saturation metrics
- Develop cost-attribution dashboards mapping accelerator spend to workloads/teams
- Design reliable, high-throughput data pipelines (Airflow, Cloud Composer, Dataflow)
- Manage complex data ingestion and preprocessing for telemetry data
- Advise on cloud storage architecture for large-scale telemetry data
Requirements
- Strong GCP data stack experience (BigQuery, Dataflow, Cloud Composer)
- Proficiency in SQL and pipeline orchestration tools (Airflow)
- Experience building distributed, high-throughput data pipelines
- Understanding of AI/ML data requirements
- Advanced knowledge of cloud storage systems
Nice to have
- Experience with real-time streaming telemetry systems
- FinOps/cost-attribution dashboard design experience
