Lead AI Engineer with Databricks, Agentic AI

We are seeking a Lead AI Engineer with expertise in Databricks and Databricks Agentic AI processes to join our team. The successful candidate will play a crucial role in delivering agentic solutions within our Trinity data platform for PLM use cases. This position requires a seasoned professional with a strong background in AI engineering and a track record of leveraging Databricks for building agents, fully embedded within the PLM team to design, build and operate the core intelligence layer of the client's Product Lifecycle Management solution.

Responsibilities

• Embed fully within the client's PLM team from day one to understand supplier specification workflows, test method structures and properties management before writing a line of code
• Design and implement the document ingestion pipeline using Databricks ai_parse_document for PDF, Excel and Word supplier specification files, storing artifacts in Unity Catalog Volumes
• Build and maintain Agent Bricks (Information Extraction) pipelines to extract structured fields from supplier documents against the canonical PLM data model
• Develop and optimize LLM-powered grouping and recommendation logic using ai_query and Databricks Model Serving, to ensure outputs are explainable and auditable
• Build the Databricks Apps front-end (React + Python backend), including file upload, metadata grids, dropdown-driven forms, human-in-the-loop review/approval screens and write-back to Lakebase
• Integrate the Graph Database layer to store and traverse relationships between specifications, properties, test methods and AI recommendations
• Configure and maintain Azure Private Link / Private Endpoint connectivity, including DNS forwarding for databricksapps.com, to ensure all traffic stays within the client's corporate network perimeter
• Implement structured logging, lineage tracking in Unity Catalog, and audit trails to support Responsible AI governance requirements
• Contribute to ARB artefacts and technical design documentation as required
• Collaborate daily with the Business Analyst to translate PLM domain requirements into working data models and pipeline logic, and with the Delivery Manager to surface risks and effort estimates

Requirements

• 5+ years of experience in AI/data engineering, with expertise in Databricks, including Unity Catalog, Volumes and Model Serving
• Proficiency in ai_parse_document and Agent Bricks for document ingestion and information extraction
• Skills in Python, including PySpark, pandas and FastAPI
• Expertise in React for front-end development, including hooks and REST integration
• Familiarity with Azure networking, including Private Link, Private Endpoints and VNet
• Competency in LLM prompt engineering and ai_query
• Knowledge of Graph Databases such as Neo4j or equivalent
• Background in Delta Lake / Lakebase and REST API design and integration
• Understanding of Azure DevOps (ADO)
• English proficiency at B2 level or higher

Nice to have

• Background in GxP or regulated manufacturing data environments
• Familiarity with the client's Trinity AI governance pattern
• Experience with PLM or ERP systems (SAP, Teamcenter, Enovia)
• Databricks certification (Data Engineer Professional or ML Professional)
• Exposure to graph data modeling and lineage use cases