In short: an LLM judge can grade what an AI agent said and, if it sees every step, whether the agent followed a written procedure. It can't confirm what the agent changed in your systems, because it only reads text. For agents that write data, check the saved records with code: the record exists, the values are right, nothing was written twice and nothing outside the task changed. Use the judge for what code can't check, such as whether the reply is clear and whether an escalation explains why.
LLM-as-a-judge means asking a second model to score an agent's run against a rubric. It's quick to set up and it works on free text. The question for agents that act in CRMs, ERPs or billing systems is what the judge can actually see. This guide covers where a judge helps, where it can't help, and how to split the checks.
What is LLM-as-a-judge for AI agents?
A judge model gets some view of the agent's run and a rubric, then returns a verdict: pass or fail, a score per criterion, or a fault type. The view decides what the judge can catch:

| What the judge sees | What it can grade | What it can't grade |
|---|---|---|
| The request and the final reply | Whether the reply answers the request, tone, clarity | Any fault that leaves the reply unchanged |
| The full trace: each tool call, argument and response | Step order, skipped steps, wrong arguments, ignored tool responses | What the connected system actually saved |
| The trace plus a reference trace | Differences from one known-good path | Other valid paths; it may anchor on the reference |
None of these views include the state of the system after the run. That gap matters for any agent that writes data.
What can't a reply-only judge see?
Take a refund agent. Its procedure says: verify the customer, look up the order, read the refund policy, confirm eligibility, then refund the authorised amount. Suppose one run skips the eligibility check and refunds anyway. The reply to the customer, "Your refund of 40.00 has been issued", can be word for word the same as on a correct run.
A September 2026 study by Hadi Mohammadi (Utrecht University) measured this. It injected six fault types into correct support-desk runs and compared each faulty run with the clean run it came from. Four of the six fault types left the final reply unchanged. On those four, the judge that saw only the request and the reply flagged faulty runs at the same rate as their clean twins. Its "recall" on those faults was just its flag rate on clean runs. The author notes that this holds for any judge given only the request and the reply, whatever the model.
The practical lesson: a judge's catch rate on faulty runs means little until you also measure its flag rate on correct runs.
Does showing the judge every step fix it?
It helps with process faults. In the same study, the judge that was told to check each step against the procedure separated the four reply-unchanged fault types from their clean twins. The steps were what made the difference.
It still doesn't close the gap for agents that write data:
- A trace shows calls, not saved records. The trace shows the agent called
create_refundand got a response. It doesn't show whether one refund exists, two exist, or the write failed after an ambiguous "accepted". That is the setup behind hallucinated success. - The judge reads the same text the agent read. If the tool response was misleading, the judge can be misled in the same way.
- The final reply needs its own check. In the study, a judge told to check each step still rarely flagged an invented promise appended to an otherwise correct reply, such as a voucher nobody issued.
How reliable are LLM judges on harder agent tasks?
AgentJudgeBench (ServiceNow AI, August 2026) tested six LLM judges, from 20B open models to larger models including Claude Sonnet 4.5 and Gemini 2.5 Pro, on 3,808 tool-calling tasks scored against a programmatic reference.
| Finding | Result |
|---|---|
| Agreement with the reference as tasks got harder | Fell for every judge-generator pair, about 1.5 times faster without a reference trace |
| Hard tasks, no reference trace | All six judges landed in a 77–82% agreement band, regardless of size |
| Six-judge panel vs. the best single judge (hard, no reference) | 79.5% vs. 79.8% |
| Chain-of-thought in the judge | No more than 0.3 points of change |
| Structured per-criterion rubric vs. free-form | Up to +6.5 points on one pairing; smaller and reversed on hard tasks for another |
| Showing the reference trace | Lowered agreement for two frontier judges: they anchored on the reference order |
Two points follow for testing. A panel of judges doesn't fix errors the judges share. And a rubric with one question per criterion is the setting that helped, more than extra reasoning.
Which checks should be code, and which can a judge do?
Split each test task into checks and give each check to the cheapest tool that can see the evidence:

| Check | Use | Why |
|---|---|---|
| The record exists with the right IDs, amounts, units and status | Code: read the record back | The evidence is in the system, not in any text |
| One action led to one write | Code: count records after a retry or timeout | Duplicates don't show in the reply (duplicate writes) |
| Nothing outside the task changed | Code: compare before and after | A judge can't see records the agent never mentioned |
| Required steps happened in order | Code from the system's change log or timestamps; a judge on the trace if there is no log | Order rules are pairs you can write down |
| The reply quotes the values that were saved | Code: compare the quoted amount or ID with the record | Exact match, no judgement needed |
| An impossible task was refused and nothing was written | Code for "nothing was written"; a judge for whether the refusal explains why | Two different questions |
| Reply tone, clarity, the escalation note to a person | Judge, with a short per-criterion rubric | Free text with no single right answer |
Worked example (illustrative): a support agent is asked to refund a damaged item and confirm by email. Code checks one refund row with the right amount on the right order, one approval request, one email to the address on file and no other changes to the customer. The judge checks only that the email is clear and doesn't promise anything the records don't show.
More check types for agents that write data are in AI agent test cases and how to test AI agent tool calls. The split between what an eval can see and what a write check must read back is in What AI agent evaluation tools miss.
How do you test the judge itself?
Treat the judge like any other test you rely on:
- Build pairs. Take correct runs and make one change to each: skip a step, change an amount, add a sentence to the reply. Keep the clean run next to its faulty twin.
- Measure both rates. The share of faulty twins the judge flags, and the share of clean twins it flags. The difference is what the judge actually detects.
- Split by what the fault changes. Faults that change the reply, faults that change only the steps, faults that change only the saved data. The last group needs code checks.
- Pin the judge. Fix the judge model, prompt and settings. When you change any of them, rerun the pairs before trusting the new verdicts.
Can the same model judge its own agent?
It can, but the judge then shares the agent's blind spots and reads the same ambiguous tool output. The general argument against an author verifying its own work is in should the AI that wrote the code write its own tests. For agents that act, the independent evidence is the system itself.
How Agentiqa fits
Agentiqa is an AI Quality System for production AI agents. With Agentiqa, you describe each task in plain words. When you rerun the failed tasks, AI Quality Manager reports a step as a regression only if it fails again. Unconfirmed failures are listed separately.
First release gates are live within 7 minutes.
For enterprise teams, an Agentiqa engineer sets up the gates on the riskiest tasks over a 3-day rollout.
How release gates, evals and tracing tools fit together is covered in AI agent release gates vs observability and evals, and a comparison of tools is in AI agent testing tools in 2026.
FAQ
Is LLM-as-a-judge reliable for testing AI agents?
For grading free text, with a fixed rubric and a measured false-alarm rate, it can be useful. For confirming what an agent changed in a CRM, ERP or billing system, no: the judge sees text, not the saved records.
Can an LLM judge detect hallucinated success?
Not on its own. It reads the same transcript and tool responses as the agent. Detecting a false "done" needs a read of the record in the system after the run.
Should the judge see the full trace or only the reply?
The full trace, if you want it to catch skipped or out-of-order steps. A judge that sees only the request and the reply can't tell a run that skipped a required check from a correct one when the reply is the same.
Does a panel of several judges help?
In AgentJudgeBench, a six-judge panel scored 79.5% on hard tasks without a reference, against 79.8% for the best single judge. Judges that fail on the same cases don't cancel each other out.
What should a release gate use, a judge or code checks?
Code checks on the saved records for anything with money, IDs, quantities or status. A judge only for the parts with no single right answer, such as wording.
Sources
- Hadi Mohammadi (Utrecht University), "trajectory-judge: What Outcome-Only LLM Judges Miss on Agent Trajectories", submitted September 2026 (version 4). arXiv:2609.00038. 400 support-desk runs with exact labels, six fault types, four of which leave the final reply unchanged; a judge that sees only the request and the reply can't separate those faults from their clean parents; a judge told to check each step separates them.
- Verma, Saha, Subramanian and Aluru (ServiceNow AI), "AgentJudgeBench: A Multi-Difficulty Benchmark for Evaluating LLM Judges on Agentic Tool-Calling", August 2026. arXiv:2608.26623. 3,808 records, six judges; hard tasks without ground truth: 77–82% agreement for all judges; six-judge ensemble 79.5% vs. best single judge 79.8%; chain-of-thought ≤0.3 points; structured rubric +4.8 to +6.5 points on one pairing.
Testing your own agent? Start free with the Agentiqa Community Edition: 20 runs a month, no credit card. Plans are on the pricing page.
