Every agent needs a judge

Short answer: an eval is a repeatable test that scores an agent's output. A judge is the gate that uses that score to decide what ships. Without both, you find out an agent is wrong when a customer tells you.

The failure nobody sees

Agents rarely fail loudly. They return a tidy, well-formatted answer that happens to be wrong. A job that returns an empty result and reports success is the same failure in a different form.

If nothing scores the output, wrong and right look identical in your logs.

Two kinds of check

Use both. They catch different things.

Deterministic checks. Code that verifies facts: the price is a number, the licence ID exists, the date is in the future, the total matches the line items. Cheap, fast and never moody.
A model judge. A second model scores what code cannot: is the copy accurate to the source, is the tone right, does the summary leave out something that matters. It works from a written rubric, not a vibe.

Building the gate

Collect 50 to 200 real examples of the task, with the answer a good employee would give.
Write the rubric. If two of your people would disagree on a score, the rubric is not done.
Run the agent on the examples and score every output.
Set a threshold. Output above it ships. Output below it goes to a person.
Keep the examples. Every change to a prompt or a model gets re-run against them before it goes live.

What the score buys you

It tells you when the agent is ready, in a number your finance lead can read. It tells you when a model upgrade made things worse. It lets you send only the hard cases to people, which is where their time is worth the most.

When the judge degrades

Judges drift too. If scores slide, check the deterministic results first. If those held steady and the judge moved, the judge changed. If both moved, the agent did.

An agent you cannot score is an agent you cannot improve.