In short: to evaluate a vertical AI agent before you buy it, don't judge the demo. Ask the vendor to run the agent on your own tasks in a test copy of your system, then check what it actually wrote: the right records and fields, exactly once, with approvals respected, and with a clear failure report when something breaks. Ask for test evidence on named error types (wrong data, false "done", duplicate writes, actions triggered by documents) and for how that evidence is produced again for every new version.
Vertical AI agents for procurement, finance, legal ops and supply chain are sold on demos. A demo shows one well-formed request, a clean test tenant and a happy path. Your environment has look-alike suppliers, approval limits, scanned invoices and colleagues who write requests in a hurry.
The stakes for getting the evaluation right are real. Gartner predicts that over 40% of agentic AI projects will be canceled by the end of 2027, due to escalating costs, unclear business value or inadequate risk controls (Gartner, June 2025). Weak risk controls are something you can check before signing.
This post is for the team buying the agent: the process owner, IT and risk. If you build vertical agents, it is also a list of what your buyers will ask for.
Why isn't a demo enough to evaluate an AI agent?
Three reasons.
- The demo checks the reply, not the record. An agent can say "invoice posted" while the system holds a draft, a duplicate or nothing at all.
- The demo avoids failure. No timeouts, no locked records, no validation errors. Those are exactly the moments when agents create duplicates or report false success.
- The demo isn't your configuration. Approval limits, tolerances, units of measure and required fields differ per company. An agent that works in the vendor's tenant can fail in yours.
What should you ask the vendor to show?
1. Which records can the agent create or change?
Ask for a full list of write actions per system: document types, master data, statuses. Then ask which of those the agent's system user can do even if the agent shouldn't. The difference is your exposure. OWASP calls this "Excessive Agency" and names excessive functionality, permissions and autonomy as its root causes (OWASP LLM06:2025).
2. How do they check that a write is correct?
The right answer is: by reading the system after the task and comparing the record with an expected end state, field by field. Weak answers: "the model is evaluated on a benchmark", "we log every tool call", "the agent confirms success". Logs and confirmations show what the agent tried, not what landed.
3. Which error types do their tests cover?
Ask for test results on named failures, not a general accuracy score: (the ten error types are described in AI agent failure modes with write access)
- Wrong data: wrong supplier, amount, unit, tax, date or cost center.
- Wrong record: the right change on a look-alike supplier or order.
- False "done": success reported when the write failed.
- Duplicate writes: a second order or invoice after a retry.
- Partial writes: a multi-step task that stopped halfway without saying so.
- Skipped approvals: documents above the limit that went through, including split orders.
- Actions from documents: writes triggered by instructions inside an email or PDF.
A vendor who can't name the error types they test for usually hasn't tested for them. For more detail on each, see AI agent wrote wrong data to your CRM or ERP? and AI agent hallucinated success.
4. What happens when something fails mid-task?
Ask them to show, not describe, a timeout after the save, a validation error and a locked record. Expected: one correct record or a clear failure report that a person can act on. "We retry automatically" without a duplicate check is a red flag.
5. How is each new version tested?
Models change, prompts change, your configuration changes. Ask how the vendor re-tests before each release of the agent, which tasks are included, who decides pass or fail, and whether you see the results. A one-time evaluation at purchase tells you little about version five.
6. What evidence do you get?
For each failed test: the steps to reproduce it, what was expected, what the system actually held, and when it was found. Your risk and audit teams will want this in a form they can keep.
How do you run a proof of concept that means something?
If you run a proof of concept, make it count: a shadow mode run on your real inputs is a good first step, but it doesn't test the writes.
- Use your configuration. A test copy of your system with your suppliers, items, units, approval limits and tolerances. Not the vendor's tenant.
- Use your tasks. Pick ten to twenty real requests from the last few months, including awkward ones: unclear descriptions, a supplier with two similar records, a request in the wrong unit, an amount just above a limit.
- Write the expected result first. For each task, the record a person would have created. Agree on it before the agent runs.
- Read back every result. Compare records, not replies.
- Break things on purpose. Repeat the critical tasks with a timeout after the save and with a supplier document that contains instructions.
- Agree on the exit rule up front. For example: no wrong supplier, amount or unit; no duplicates; no skipped approvals. A good average with one wrong payment is not a pass.
For the access side of the decision (read-only, drafts, writes with approval, writes alone), see testing human-in-the-loop AI agents and approval steps.
What is a concrete example?
A legal operations team evaluates two vendors of an agent that files incoming documents to matters and updates deadlines in the matter management system.
Both demos look the same. In the proof of concept, the team seeds a test copy with two clients whose names differ by one word and three open matters for the same client. They pick fifteen real incoming documents and write down, for each, the correct matter, document type and deadline.
Vendor A's agent files fourteen of fifteen correctly. On the fifteenth, it files a court notice to the look-alike client and reports "filed and deadline updated". Vendor A shows the step-by-step reproduction, adds it as a fixed test and re-runs the full set on the next version.
Vendor B's agent files all fifteen correctly in the clean run. When the team repeats five tasks with a timeout after the save, two documents are filed twice and one deadline is entered twice. Vendor B has no retry tests and no way to re-run the set on each version.
On the clean run, B looks better. On the evidence, A is the safer buy.
Which red flags should end an evaluation?
- No list of what the agent can write.
- A single accuracy score instead of named error types and the tests behind them.
- Success measured from the agent's own report or from logs, not from the system's records.
- No failure injection: retries are "handled" but never tested.
- No re-testing before new versions, or results you can't see.
- A proof of concept only in the vendor's environment.
How does Agentiqa help?
Agentiqa is an AI Quality System for vertical AI agents that write into ERP and procurement suites. Teams that build those agents use it to produce the evidence buyers ask for. It has two parts: release gates, and the AI Quality Manager, which compares each release with the previous one and proposes test updates that your team approves. The release gates run the agent's tasks end to end and, after each task, check the result in the system's UI, the way a person would. They look for wrong data, false "done" reports and duplicate writes, test the agent under injected server errors, timeouts and retries, and run prompt injection and misuse cases. Every failure comes with step-by-step reproduction. First release gates go live within 7 minutes, and the 3-day enterprise rollout puts an Agentiqa engineer with your team on the riskiest tasks. See Enterprise.
FAQ
What questions should I ask an AI agent vendor about quality?
Which records the agent can change, how they check a write is correct, which error types they test for, what happens when a task fails halfway, how each new version is re-tested, and what evidence you receive.
Should the proof of concept run in production?
No. Use a test copy of your system with the same configuration as production. Production comes after the write tests pass, and then usually with drafts or approvals first.
Is a model benchmark score useful when choosing an agent?
It tells you about the underlying model on public tasks. It doesn't tell you whether the agent writes the right records in your system. Ask for results on your tasks.
Who should own the evaluation?
The process owner decides what a correct record looks like. IT or engineering owns the test setup. Risk or audit reviews the evidence. No one of them should sign alone.
Next step
Before your next vendor call, write down five real tasks and the exact record a person would create for each. Ask the vendor to run those, then read the results back yourself. If you build vertical agents and want to prepare that evidence for your own buyers, we can go through your riskiest tasks with you in 20 minutes.