Lead ML/AI Solution Engineer with Databricks

We are seeking a seasoned Lead ML/AI Solution Engineer to architect, develop, and launch a personalization platform that delivers recommendation and ranking models via a scalable API layer. The system comprises a Python FastAPI service exposed through AWS API Gateway and Lambda, integrating with Databricks Model Serving endpoints.

The engineer will partner closely with Data Scientists to operationalize personalization models, guarantee production-grade reliability, and confirm performance through testing and monitoring.

Responsibilities

• Partner with Data Scientists to adapt, package, and deploy personalization models (recommendation and ranking models) to Databricks Model Serving using MLflow
• Develop and maintain a FastAPI service that invokes one or more model serving endpoints
• Apply production-grade engineering practices, including caching, connection pooling, request timeouts, and fail-open fallback mechanisms
• Set up secure authentication between services and Databricks using Service Principals
• Establish and maintain the AWS integration layer, including API Gateway, AWS Lambda, and networking/egress connectivity
• Define and govern infrastructure using Terraform and build a simple CI/CD pipeline
• Enable observability and monitoring, including latency, error, and fallback metrics
• Run load and performance testing to validate system behavior under expected traffic volumes
• Specify REST API contracts and implement contract testing to enable early integration by consuming teams
• Carry out offline evaluation of personalization models and support online validation through A/B testing or shadow testing where feasible
• Present technical results and business impact to both technical and non-technical stakeholders

Requirements

• 5+ years of experience in ML/AI engineering, building and serving recommendation or ranking models, including offline model evaluation
• At least 1 year of relevant leadership experience
• Hands-on knowledge of Collaborative Filtering, Two-Tower Architectures, and Learning-to-Rank approaches
• Strong proficiency in Python with production experience building services using FastAPI
• Understanding of asynchronous processing, connection pooling, and caching strategies
• Expertise in timeout handling, fail-open mechanisms, and resilient service design
• Background in registering, versioning, and serving ML models with MLflow and Databricks Model Serving
• Familiarity with Service Principal authentication, endpoint scaling, and version management
• Competency in monitoring latency, errors, and fallback rates, including creating alerts and dashboards
• Skills in load-testing tools such as k6 or Locust, with the capability to validate p99 latency targets
• Capability to explain technical concepts and outcomes in business terms, working with engineering, data science, and business stakeholders
• English proficiency at an Upper-Intermediate level (B2) or higher

Nice to have

• Proficiency in Infrastructure-as-Code with Terraform and CI/CD pipeline implementation adequate to deploy and hand over a reproducible POC environment
• Familiarity with AWS API Gateway and Lambda integration, including awareness of cold starts and performance optimization
• Knowledge of VPC networking and egress connectivity to Databricks
• Expertise in OpenAPI-first REST API design and contract testing with Pact or schema-based testing approaches
• Capability to architect services for multi-region deployment and future scalability
• Understanding of GenAI for personalization, including embeddings, vector search, LLM-based re-ranking, and personalized explanations powered by GenAI