AI Evaluation Engineer Mid · Full-time · Remote

You will build automated evaluation benchmarks, hallucination detection suites, and quality scoring systems for client AI models.

As an AI Evaluation Engineer at TheWebVale, you will ensure our AI applications are accurate, consistent, and safe before they reach client users. You will construct automated evaluation datasets, grading rubrics, and regression test suites for healthcare, legal, and coaching tools. You will collaborate with AI engineers and client subject matter experts in our two-to-three week release cadences. You will prevent prompt regressions, detect hallucinations, and provide data-backed confidence scores for every AI release.

AI Evaluation Engineer

Python

AI Evaluation

Ragas

API Testing

FastAPI

Git

Responsibilities

Your day to day.

6 things, one at a time. Keep scrolling.

• Design comprehensive evaluation datasets representing edge cases and common user prompts.
• Build automated testing harnesses that grade model outputs using heuristic and LLM judges.
• Track accuracy, factual consistency, relevance, and safety metrics across prompt iterations.
• Implement automated regression testing in CI pipelines before deploying prompt changes.
• Analyze model failures and collaborate with prompt engineers to refine system instructions.
• Generate readable evaluation reports showing accuracy improvements over project iterations.

Requirements

What we look for.

6 things, one at a time. Keep scrolling.

• Hold 2–4 years of software engineering or data analysis experience with AI testing focus.
• Demonstrate high proficiency in Python, Pandas, regular expressions, and API testing.
• Show practical experience with evaluation frameworks like Ragas, DeepEval, or Promptfoo.
• Understand statistical sampling, inter-annotator agreement, and qualitative grading rubrics.
• Possess analytical rigor with zero tolerance for undetected model hallucinations.
• Communicate evaluation outcomes and reliability risks objectively to project stakeholders.

What you get.

thewebvale.

• Remote-first, with a Bhopal studio for hybrid days
• Real client products across many industries
• Direct work with the founder and senior team
• Learning and tooling support
• Your work featured in our public portfolio

How we hire

Three steps, no tricks.

• Intro chat

A short call about what you want to work on and how you like to work.

• Work review

Walk us through projects, code or designs you are proud of.

• Paid task

A small paid task or pairing session, so both sides see the fit.