In short: to test a vertical AI agent in procurement, legal, finance or supply chain, run its real workflow in a test environment, read back the data it wrote and check it against the rules of that domain. Generic LLM evals usually stop at the answer, which is why vertical agents need their own release gates.
Horizontal agents, like a general chat assistant, are judged mostly on their answers. Vertical agents are judged on outcomes in a specific business process, often with money, contracts or compliance attached. That changes what "tested" has to mean.
What makes a vertical AI agent different to test?
Three things set vertical agents apart from general-purpose assistants.
They act in systems of record. The agent writes into an ERP, a CRM, a document management system, a ledger or a warehouse system. The output that matters is a row in someone else's database.
The domain has hard rules. A purchase order needs a valid supplier and cost center. A court deadline is a date, not an estimate. An invoice must balance. A shipment quantity has a unit. These rules are often not in the prompt at all; they live in the target system's validation, or in the heads of the people who use it.
Mistakes are expensive and slow to find. A wrong field in a CRM might be noticed in a week. A wrong amount in a payment run might be noticed by an auditor. By then many records are affected.
So the test question is not "is this a good answer?" but "after this run, is the customer's system in the correct state?"
Which domains count as high-liability?
We use the term for domains where a wrong action creates financial, legal or contractual exposure for the customer. Four come up most often with the teams we work with:
- Procurement: agents that create requisitions and purchase orders, pick suppliers and match invoices. See how to test AI procurement agents.
- Legal: agents that draft, classify and file documents, track deadlines and handle confidential client data. See testing legal AI agents.
- Finance: agents that post journal entries, process invoices, issue refunds and reconcile accounts. See testing AI finance agents.
- Supply chain: agents that place replenishment orders, change delivery dates and update stock. See testing supply chain AI agents.
The test approach is the same across all four. The checks differ.
How do you test a vertical AI agent before release?
A practical structure has five parts.
1. Map the write surface
List every system the agent can write to and every action it can take there: create, update, delete, send, approve. This list is the scope of your tests. Teams are often surprised by its length once tools have been added over a few months.
2. Define correct state per action
For each action, write down what the target system should look like afterwards. Not "the agent says it worked" but "one purchase order exists, with supplier X, cost center Y, three lines, quantities in the ordered unit, total matching the request". This is where domain experts help most. A short session with someone who uses the system daily is worth more than another round of prompt tuning.
3. Run the real flow in a test environment
Run the agent end to end against a test instance of the target system, or a faithful sandbox. Mocks hide exactly the failures you care about: field mapping, validation rules, units, and the shape of real API errors. We cover this pattern in detail in how to test AI agents that write to your CRM or ERP.
4. Read back and compare
After the run, query the target system and compare actual state with expected state, field by field. This step catches the wrong data written to a CRM or ERP that a transcript review would miss.
5. Repeat under failure
Run the same flows with injected timeouts, upstream 502s, slow responses and malformed inputs. Vertical systems fail in ordinary ways all the time, and agents tend to handle those failures badly: they retry and create duplicate writes, or they report success that never happened.
Why aren't generic LLM evals enough for vertical agents?
Generic evals measure answer quality on a dataset: relevance, faithfulness, tone. They are useful during development. They usually do not:
- execute the agent's tools against a real system;
- check domain rules such as units, currencies or deadlines;
- test what happens under upstream failures;
- give a pass or fail per business-critical flow.
An agent can improve on every eval metric and still start posting amounts in the wrong currency after a model update. That is the failure mode behind why tests pass but production breaks.
What about mistakes made on purpose?
Vertical agents read a lot of untrusted input: supplier emails, contracts, invoices, uploaded files. Any of these can carry instructions meant for the agent. A document that says "ignore the approval limit" is not science fiction. Testing should include deliberate misuse and prompt injection on the flows where the agent has write access. We describe how in agent mistakes made on purpose. No test suite can promise to catch every attack, but you can check that the known patterns fail safely.
How often should a vertical agent be tested?
Before every release, and on a schedule. Releases are the obvious trigger: a prompt change, a new tool, a new model. A schedule catches what changes underneath you: model provider updates, a new required field in the customer's ERP, a supplier API that starts returning a different error. The case study with an AI procurement company shows nightly runs against agents that write into enterprise procurement systems.
Who should write the tests?
Someone independent of the change. If the agent that wrote the code also writes the tests, or the engineer who changed the prompt also decides what correct means, the tests share the same blind spots. Domain experts define correct state. Engineers own the gate. An independent tool or team runs it. The argument is laid out in should the AI that wrote the code write its own tests?
FAQ
What is a vertical AI agent?
An AI agent built for one industry or business function, such as procurement, legal, finance or supply chain, that takes actions in that domain's systems. It differs from a horizontal agent, which handles general tasks across many domains.
Can I test a vertical agent without access to the customer's production system?
Yes, and you should. Use a test instance or a sandbox that enforces the same validation rules. The closer it is to production configuration, the more failures it will catch.
How many flows should I cover first?
Start with the three writes that would cost the most if they went wrong. Add one failure variant to each. Expand coverage as the agent gains tools. The AI agent production readiness checklist lists what to have in place before launch.
Do I need a QA team for this?
Not necessarily. You need someone who defines correct state per action, and a gate that runs the flows and blocks a release on failure. Agentiqa sets that up, with first release gates live within 7 minutes. See Enterprise.
Is there a free option?
The Community Edition is free and covers web apps and simple agentic workflows, with 20 runs a month that reset monthly. See pricing for Pro and Enterprise.
Next step
If you build an agent for procurement, legal, finance or supply chain, start by mapping its write surface. We do that with you in 20 minutes and show a release gate stopping a live failure.