For well over a year, our AI agents have been testing web applications end to end — hundreds of them, from approval workflows to our clients' own AI products. Here's the honest summary of that year: getting an AI to drive a browser stopped being the hard part long ago. The hard part — the part we still hunt bugs in, deliberately, night after night — is the verdict: the moment a system says pass or fail and you act on it.
The hunting happens on our proving ground. Carmakers crash their own cars on a private track so that no customer ever crashes one on the road. We do the same with verdicts: realistic business flows — approval chains, document uploads, multi-step forms — rebuilt as replicas on our own infrastructure and replayed over and over, with one goal: break our own verdict layer before anything reaches a customer.
This July, the hunt brought back two textbook specimens in a single week.
Night one: the application worked perfectly. The agent logged in, filled the forms, walked the approval flow to exactly the screen it was supposed to reach — every step genuinely succeeded. The test came back red.
Night two: a price on the screen didn't match what the test expected. Anyone looking at it would have said "that's wrong." The test came back green.
Both times, the browser part did its job perfectly. Of course it did — that part is solved. What broke, in opposite directions, was the verdict. And that's the story of this article: why the verdict layer is where agentic QA is actually won or lost.
If you're new to the category, start with our primer, What is Agentic Testing? This article is about the next question — the one that matters now that every agent can drive your app: why should you believe what it tells you?
Driving the browser is a commodity. The verdict is the product.
Let's give the browser part its due — by admitting it's no longer special. Every modern coding agent can open a browser and click through your app today: Claude does it, ChatGPT's agents do it, half the tools on your desktop do it. "AI + a browser" isn't a demo anymore, and it isn't a moat. It's a commodity — you can have it this afternoon, often for free.
And that's precisely the point. When a capability becomes a commodity, it stops being the product. If agentic QA were just "an AI that clicks around," the category wouldn't need to exist — you'd already have it.
One honest nuance, though — because this is where the commodity comparison quietly breaks down. A coding agent looks at the page only when it needs to figure out the next click. A QA system has to look at every step: capture what's actually on screen, analyze it, and keep it — because that's the evidence the verdict will be judged on later. Ask a general-purpose agent to do this and you hit two walls at once. By default, it simply doesn't — it glances, it doesn't document. And if you insist, the bill explodes: full screenshot-and-page analysis, on a frontier model, at every step of every run of every test. Making that observation affordable — a pipeline and models tuned for exactly this job — is months of unglamorous engineering, and it's the first thing that separates can click through your app from can testify about what your app did.
So what's left? Think of every one of those agents as a fast, tireless intern with a mouse. Everyone has one now. The question stopped being can the intern click years ago. The question is: whose intern's report would you sign a release on?
That's what a QA verdict really is: a claim you act on. A red verdict blocks a release, pages an engineer, interrupts someone's evening. A green verdict lets code ship to real users. Get either one wrong and there's a bill — and the two bills are very different.
What does a wrong verdict actually cost?
A false alarm — red on a working feature — feels harmless. It isn't. The first time, your team investigates for an hour and finds nothing. The second time, they investigate for ten minutes. The third time, they stop looking. It's the boy who cried wolf, running inside your CI: after a few false alarms, red stops meaning anything, and then the whole system is decoration. In our July specimen, the flow was fine and the verdict wasn't — on the proving ground, that's a catch; in a customer's CI, it would have been a fire drill. Nothing erodes trust in QA faster.
A silent pass — green on a real problem — is the opposite bill. Nobody investigates anything, because everything looks fine. The wrong price ships. Weeks later a customer finds it for you, in production, at the worst possible moment.
Here's the asymmetry that shapes everything: false alarms burn your trust immediately, while silent passes cost you later. A serious agentic QA system has to treat both as first-class bugs. A commodity browser agent treats neither — nobody benchmarks whether its verdicts can be trusted. They benchmark whether it clicked.
Why can't the AI just check its own work?
Back to the wrong price that passed. How does that happen?
Simple: the agent that performed the test also graded it. Somewhere along the way it decided the run "looked successful" — and once the doer becomes the grader, confidence beats evidence.
We've written about this exact principle one level down: should the AI that wrote the code write its own tests? The answer doesn't change when you move up a level. The AI agent that runs your test must not be the one grading the run.
The way out is a division of labor:
- The AI does what it's genuinely great at: reading the screen. "What does the price field say?" — no opinion attached.
- A separate, deliberately boring, deterministic check does the judging: does what's on screen match what the test expects — yes, no, or can't tell?
The AI extracts. The rules decide. The one thing the AI never gets to do is approve its own homework.
Word-matching or fact-checking?
Here's a tiny example that explains half of all verdict failures.
A test expects the supplier's country to read "Germany." The screen shows "DE."
A human looks at this and sees the same thing twice. A byte-by-byte comparison sees a failure — and fires a false alarm on a perfectly working feature.
So just compare "smarter," right? Careful. Meet "CA." Is that Canada — or California? Is "Turkey" a country or a bird? Loosen the matching and you stop catching real mistakes: now wrong values slide through because they look vaguely similar. You've traded false alarms for silent passes, which is not a trade — it's the same disease in the other direction.
The same trap hides in where you look, not just what you compare. We once saw a check for "status = Approved" pass because the word "Approved" existed somewhere on the page — in a different panel, about a different thing. Finding a word is not the same as verifying a fact.
The way out is what we call fact-checking instead of word-matching:
- Word-matching asks: do these letters match?
- Fact-checking asks: is this the same fact — the right value, of the right kind, in the right place? "DE" equals "Germany" because country codes are rules, not vibes. "CA" alone proves nothing, and an honest system says so instead of guessing.
That last part matters more than it sounds. A trustworthy verdict system needs three answers, not two: pass, fail, and "couldn't confirm — and I'm telling you." Take the third answer away, and you force the system to guess. Guessing is where both false alarms and silent passes are born.
The math nobody does out loud
Vendors in this space love per-step accuracy. One published benchmark proudly reports that the agent's step-level verdicts — its individual checks — are right more than 85% of the time (Smartesting).
Sounds great. Now do the multiplication.
One end-to-end test run isn't one check — it's many. The agent verifies something after almost every move: logged in? form saved? status changed? total correct? A realistic run easily makes ten checks. If each check's verdict is 85% reliable, the chance that every check in the run is graded right is 0.85¹⁰ — about 20%. And one wrongly graded check is all it takes to flip the whole run into a false alarm or a silent pass. Independent testing of whole browser-agent tasks points the same direction: under 50% single-run success on longer tasks, dropping below 25% when the same task is repeated eight times.
Which exposes the quietest failure of all: run the same test twice on the same build and get two different verdicts. That's not a verdict. That's a coin.
None of this means agentic QA is hopeless. It means per-step cleverness doesn't compound into trust on its own. Repeatability is not an AI feature — it's something you have to engineer around the AI: rules that decide, evidence that's recorded, runs that can be replayed and compared.
What separates a QA system from a browser agent
Both July specimens were fixed, re-verified on the proving ground, and turned into permanent regression tests within about 48 hours. But the fixes weren't the lesson. The lesson is a checklist a year of this hunting has produced — the one we now treat as the definition of the category. A real agentic QA system has:
- Three answers, not two. Pass, fail, and an honest "couldn't confirm." No guessing dressed up as confidence.
- Separation of doer and grader. The AI reads the screen; deterministic rules decide the verdict. The agent never approves its own work.
- Facts over words. Values are compared by meaning and by place — with rules where meaning is provable, and an honest "couldn't confirm" where it isn't.
- Repeatable verdicts. Same app, same test, same verdict. Every run leaves evidence — what was on screen, where, when — so any verdict can be replayed and audited.
- False alarms treated as bugs. With the same severity as missed bugs, because they kill the product just as surely — only faster.
Notice what's not on the list: a smarter model, a better prompt, a fancier browser. None of it comes bundled with the commodity — every item is system engineering around the AI. That's the "more than" in this article's title, and it's roughly 80% of the work. The clicking you can get anywhere. The verdict you can trust is the product.
Five questions to ask any agentic QA vendor
Including us.
- When the agent can't confirm a check, what does it do — guess, or say so?
- Who grades a run: the same AI that performed it, or an independent check?
- If a label changes from "Germany" to "DE" tomorrow, what happens to my test?
- If I run the same test twice on the same build, do I get the same verdict?
- Can you show me the evidence behind any verdict — what was on screen, where, and when?
And a bonus question that separates marketing from engineering: how do you find out when your own verdicts are wrong? If the answer is silence, the verdicts are being graded on trust.
FAQ
Isn't agentic QA just "an AI running my tests in a browser"? That part is a commodity — any modern coding agent can drive a browser today, and it's genuinely useful. But it's the execution half, and the cheap half. The QA product is the verdict: deciding pass or fail reliably enough that your team acts on it without double-checking. An agent that navigates brilliantly but grades unreliably isn't a QA system — it's a very confident intern.
Are false alarms really that bad? Missing a bug sounds worse. They fail differently. A missed bug costs you once, later. False alarms cost you the whole system: after a few of them your team stops trusting red, and then even the correctly caught bugs get ignored. Both directions have to be engineered against — deliberately.
How does Agentiqa know its own verdicts are right? We maintain a proving ground: replicas of realistic business flows that we replay nightly and compare verdicts across repeated runs — a verdict that flips without a code change is treated as a bug in our system. Our checks return three answers, not two, and every verdict carries its evidence. And when the hunt catches a verdict gone wrong — that's what the range is for — the fix isn't complete until the proving-ground runs come back green again, repeatedly.
Ready to see the difference between a browser agent and a QA system? Point Agentiqa at your staging URL — free to start in the browser or with the desktop app — and the first verdicts take minutes, not weeks.