Senior AI QA Engineer with Python (Manual & Automation)

EPAM·Colombia, Mexico·Удалённо·сегодня

We are seeking a Senior AI QA Engineer to validate the performance of AI-driven video analysis systems, focused on detecting key moments in sports content. The role combines manual and automated testing, working with pre-labeled video assets provided by the customer.

Responsibilities

Audit live sports games (NBA, MLB, NFL, NHL) to ensure the AI/Inference Service correctly tracks and labels major sports moments such as touchdowns, home runs, and buzzer-beaters as they happen
Work with timecodes, video frames, transcriptions, and captions, applying sports-related contexts, lingo, and metrics
Act as the human expert who catches when the AI hallucinates, misinterprets a sports rule, or produces errors
Collaborate directly with AWS engineers to detail bugs and validate resolutions
Review AI-generated labels and metadata tags to ensure they make sense for the sport and meet advertising industry standards (IAB rules), keeping content brand-safe for ad placement
Compare the inference system's outputs against customer-provided labels to determine accuracy and identify missed or incorrectly detected events
Execute both manual test cases (for nuanced or edge cases) and automated test cases (for validating outputs at scale and ensuring consistency)
Log discrepancies, defects, and gaps between expected and actual results, and collaborate with the inference/development team to triage and resolve issues
Maintain clear QC documentation, track accuracy metrics, and help build a smooth testing process that bridges traditional sports broadcasting with new AI technology
Serve as a communication bridge between the customer and technical teams, clarifying results and expectations
Repeat validation iteratively as new labeled assets or model updates are provided to ensure ongoing accuracy and reliability

Requirements

3+ years of experience in both manual and automation QA, preferably in video-focused or AI-driven environments
Knowledge of testing LLMs and understanding of their workflow
Proficiency in automation scripting with Python, Selenium, or similar tools for comparing JSON outputs to ground truth at scale
Understanding of software development and QA cycles, including defect logging, triage, and reporting
Ability to interpret labeled data and validate model outputs
Familiarity with concepts like computer vision and transcript analysis (no need to understand internal model workings)
Strong communication skills for stakeholder interaction and reporting
Detail-oriented and iterative approach to testing
Proficiency in English at an Upper-Intermediate level (B2) or higher