Title: I Built My Own AI Judge — Here Is Why Every Agent Needs One

My AI agent had no quality gate.

It worked in demos. It impressed my team. Then I deployed it, and within a week, users started reporting bad answers. Hallucinations. Outdated facts. Responses that sounded right but cited framework versions that did not exist.

I tried manual review. It did not scale. I tried using Claude as a judge; it worked, but I was paying API costs just to check my own agent’s work. And the judge was not trained on my domain, my standards, and my failure cases.

So I built my own.

The Approach

I fine-tuned a DeepSeek-R1-Distill-Qwen-7B model using LoRA on FPT AI Factory’s H200 GPU Cloud. The model evaluates every agent response on four criteria: accuracy, relevance, completeness, and hallucination risk.

The training data? 200 of my own agent’s responses — scored manually, honestly, including the bad ones. Alpaca format. Each example has the question, the agent’s response, and my evaluation with reasoning.

What Surprised Me

The hardest part was not the fine-tuning. It was the dataset. Sitting down and admitting when my agent gave bad answers took discipline. But that dataset is now the most valuable asset in my entire agent pipeline.

The model caught every hallucination I tested. One response confidently claimed, “LangChain reduces development time by 60–70% based on published benchmarks.” That statistic does not exist. A generic LLM judge would probably let it pass because it sounds plausible. My fine-tuned judge flagged it immediately — because I trained it on exactly this failure pattern.

The Numbers

— Fine-tuning time: 25 minutes on one H200 GPU

— Inference: under 2 seconds per evaluation

— Total cost: under $15

— Training type: LoRA (rank 16, alpha 32)

— Dataset: 200 examples in Alpaca format

How It Works in Production

The model is deployed as an OpenAI-compatible API endpoint. I wrote a Python wrapper that takes every agent response, sends it to the evaluation endpoint, and checks the verdict. Responses below my accuracy bar get flagged for review or queued for regeneration.

My recommendation: start in logging mode. Evaluate every response, but do not block any of them. Review the scores for a week. Adjust your standards based on real data. Then enable the gate.

The Growth Model

Start with 20 hand-scored examples — your worst agent failures. At 50 examples, the model begins to generalize. At 200, you get a meaningful accuracy jump. Past 200, you hit diminishing returns unless you add new failure categories.

The dataset grows as your agent evolves. New failure patterns, new evaluation criteria, retrain the judge. The model stays current because your standards stay current.

Try It Yourself

I built this on FPT AI Factory — they offer $100 in free credits for new accounts, which covers this entire build multiple times: https://short.factory.fpt.ai/BXaZm

Full video walkthrough with every step:

This is a preview of what I am building for The Context Layer series — Part 3 goes deeper into evaluation architectures, runtime judges, and observability dashboards.

If you are building AI agents without a quality gate, you are hoping your agent works. Hope is not a strategy. Build the judge.