Lead AI Testing Engineer

We are looking for a Lead AI Testing Engineer to drive the quality strategy for AI-driven and data migration initiatives spanning knowledge graphs, ontologies and RAG-based LLM features. You will define and execute QA practices across data ingestion, graph layers, rule evaluation engines and AI/LLM components while building automation frameworks that scale evaluation across complex, data-heavy systems.

Responsibilities

• Design and automate evaluation of RAG-based and LLM-driven features including grounding, answer accuracy, determinism/reproducibility, precision, recall and hallucination rate
• Build test harnesses to scale evaluation beyond human-in-the-loop processes
• Create data-driven test suites for underwriting rules and validate rule execution via SPARQL evaluator, arithmetic/threshold logic and multi-condition scenarios
• Verify ontology schema correctness and instance accuracy against source data, perform reasoner consistency checks and SHACL validation
• Define and drive end-to-end QA strategy covering data ingestion, ontology/graph layer, rule evaluation engine, AI/LLM layer and integration with underwriting solutions
• Establish quality gates, acceptance criteria and test coverage models
• Maintain automation test suites across knowledge graph, data ontology layer, backend services and frontend layers
• Validate end-to-end flows across ontology, rule engine and decision output, perform contract testing and cross-system data consistency validation
• Integrate test suites into CI/CD pipelines and define release quality gates
• Champion automation-first and shift-left quality practices
• Leverage agentic AI and Gen AI tooling in testing and framework development

Requirements

• 8+ years of experience in software development, testing automation and DevOps
• Proficiency in Python test automation using pytest or equivalent, scripting and AI/ML library integration
• Expertise in testing Knowledge Graph or Data Ontology solutions using OWL2, RDF and SPARQL
• Skills in SHACL validation and graph databases such as GraphDB, including ontology validation, entity/relationship integrity, reasoner consistency checks and graph query correctness
• Proven ability to define and drive QA strategy independently for complex data-heavy systems, including building automation frameworks and implementing quality gates
• Skills in data validation using SQL and graph query languages such as SPARQL and Cypher, along with understanding of deterministic vs probabilistic systems
• Background in backend and API testing (REST) covering data validation, integration and E2E testing
• Proficiency in frontend web application test automation using Playwright or equivalent Python-compatible framework
• Hands-on experience using coding agents and agentic development daily
• Demonstrated experience testing and evaluating RAG-based Gen AI / LLM applications including grounding, answer accuracy and hallucination/determinism checks
• Applied knowledge of Gen AI / LLM evaluation frameworks and metrics such as precision, recall, criteria recall and efficiency, with proven ability to automate Gen AI/RAG evaluation at scale
• Experience in Agile delivery using Git and Azure DevOps or JIRA
• English proficiency at B2 level or higher

Nice to have

• Skills in semantic search testing and evaluation
• Familiarity with Vector Database integration and retrieval validation
• Background in data science covering ML concepts, data pipelines or data engineering collaboration
• Proficiency in API tooling such as Postman and Bruno
• Experience with quality-gate implementation in CI/CD pipelines such as Azure DevOps or Jenkins