Product Data Scientist — AI Evaluation & Quality (Remote)
You'll join the AI Team — the group driving all AI products and technology at Finom
We build and ship AI across the company: AI financial co-pilot, voice agent, and internal AI-powered processes
Our belief: your AI agent is only as good as your eval loop — we can build AI as good as the evals we run on it
Your mission: own that eval loop across every AI product we ship — pre-launch quality gates, post-launch monitoring, continuous improvement
You'll work directly with our AI Quality lead, Igor Kolodkin
Close collaboration with AI engineers, Product, and domain experts across the company
Core stack: Databricks, DeepEval, Claude Code
Own and extend our offline eval suite across products — datasets (capability + regression), judges, metrics
Build and maintain online quality dashboards: resolution rate, CSAT, thumbs up/down, LLM-as-judge signals, error rate, latency
Close the production feedback loop: mine failure patterns from real traffic → turn them into regression cases → propose fixes to Product and domain experts
Harden methodology: judge stability, non-determinism handling
Translate numbers into decisions – weekly syncs, clear trade-offs, no dashboards for their own sake
Python and SQL — you can build an analysis end-to-end
Solid foundation in statistics — sampling, hypothesis testing, variance, understanding what a noisy metric is
Analytical mindset — you start from the business question, not from the tool
3+ years in analyst / data scientist roles, at least one in a product context
Experience in quality analytics for ML systems — ranking, recommendations, classification, etc.
Hands-on experience evaluating LLM applications (RAG, agents, tool use, judges)
Experience building LLM agents — side projects, toy builds, personal experiments all count
AI-assisted coding is our default authoring environment, not a bonus
Claude Code is our main tool — you'll reach for it for SQL, Python, analyses, dashboards, and internal scripts
We're looking for analysts who are already curious and fluent with AI coding — or genuinely excited to become fluent fast
We care about what you ship and how clearly you think
If this idea excites you rather than worries you, you'll feel at home here
