In short: to test an AI agent that writes to a CRM, ERP or billing system, run its real write flows against a seeded sandbox, read the saved records back and compare them with what should exist, then repeat the flows with injected 502s and timeouts. Turn those checks into a release gate, so a version that writes wrong data is stopped before production.
Most guides to testing AI agents look at what the agent says: answer quality, tone, tool choice. Once an agent can create orders, update contacts or post charges, what matters is what it wrote.
What does write access change?
A bad answer from a read-only agent is just a bad answer. A bad write from an agent with write access stays in someone else's system: a truncated order ID, a contact merged into the wrong account, a duplicate charge. Someone finds it weeks later in reconciliation or a support ticket.
So the test has to prove three things:
- State, not response. "Order created" is a claim. The record in the target system is the fact.
- Failure paths, not only the happy path. Real APIs time out, return 502s and rate-limit. The expensive bugs happen there.
- Every release, not once. A one-line prompt edit or a model upgrade can change what the agent writes.
Why unit tests and evals miss bad writes
Unit tests usually mock the tools. The mock returns a clean success, and the test proves the code matches the mock. It doesn't prove the real system stored the right values exactly once.
Evals score the transcript or the final message. An eval can confirm the agent meant to order 120 meters of cable. It can't see that the ERP saved 120 pieces, or that a retry created a second order. More on this in why all your tests pass but production still breaks.
The method in six steps
- Map the write surfaces. List every endpoint, form and record type the agent can create, change or delete. Rank them by the cost of a bad write and start at the top.
- Seed a sandbox. Use a test tenant of each target system with known accounts, suppliers and products. You can only check that the agent picked the right supplier if you know which one is right. Reset the data between runs.
- Write the end state down. For each flow, say what must exist afterwards: "exactly one purchase order for request R-1042, supplier S-17, 120 meters of item C-300, status submitted." Don't prescribe which buttons or tools the agent uses; that can vary.
- Run the real flow and read back the record. Trigger the agent the way a user would, through the real integration. When it reports completion, query the target system directly and diff the saved records against the end state, field by field.
- Inject failures. Repeat the critical flows with a 502 after the write commits, a timeout, a rate limit and a 200 response with an error in the body. A correct agent ends each case with exactly one correct record or a clearly reported failure. This is where duplicate writes and hallucinated success show up.
- Gate the release. Run the suite before every release that changes to prompts, models, tools or integration code. A red result stops the release and comes with the trace, the data diff and steps to reproduce it. How to set this up is covered in AI agent regression testing before every release.
What to assert on each flow
| System | Example write | Check on the saved record |
|---|---|---|
| CRM | Create a contact and log a deal | No duplicate contact, right account, deal stage and amount match the source |
| ERP | Create a purchase order | Exact supplier and item IDs, right quantity and unit, one order per request |
| Billing | Issue a refund | Exact amount and currency, right invoice, one transaction per action |
| Any | Confirmation to the user | Every ID, amount and status matches what the system holds |
Two checks belong on every flow: one business action equals one write, and what the agent reported equals what was saved. Field-level problems like truncated IDs and unit mismatches are covered in AI agent wrong data in your CRM or ERP.
Handling non-deterministic behavior
Allow variation in how the agent works and none in what it wrote. Assert on end state, not on the sequence of tool calls. Run critical flows more than once, because a flow that fails one run in five will fail in production too. And keep the verifier outside the agent: if the agent grades its own work, it shares its own blind spots, the same reason the AI that wrote the code shouldn't be the only one testing it.
Where this sits next to evals and tracing is covered in AI agent release gates vs observability and evals.
If your agent works in procurement, legal, finance or supply chain, how to test vertical AI agents applies the same method to those domains. Inputs written to mislead the agent are covered in testing for misuse and prompt injection, and the AI agent production readiness checklist turns all of this into a list to work through before launch.
FAQ
How do you test an AI agent that writes to a CRM or ERP?
Run its real write flows against a seeded sandbox, read back the saved records and compare them with the expected end state. Repeat the flows with injected 502s and timeouts, and run the whole suite before every release.
Can I use mocks to test an agent with write access?
For unit tests of your own code, yes. For the agent's critical write flows, no. Mocks always succeed, so they never show a duplicate after a timeout or a field the real system rejects.
How often should these tests run?
Before every release that changes prompts, models, tools or integration code, plus on a schedule to catch changes in the target systems.
Shipping AI agents with write access to ERP, CRM or billing? Agentiqa sets up high-confidence release gates for vertical AI agents in high-liability domains. The gates run before every release, and the first release gates are live within 7 minutes. Book a 20-Minute Mutation Risk Audit.
Testing your own web app? Start free with the Agentiqa Community Edition.