Share
Customer System Changed? How to Re-Test Your AI Agent
November 16, 2026Read time: 10 min

Customer System Changed? How to Re-Test Your AI Agent

When a customer's ERP, CRM or procurement suite changes, a vertical AI agent can break without changing. Which changes need a re-test and how to run it.

In short: when the system a vertical AI agent writes to changes (a new required field, a renamed status, an extra approval step, a new API version), the agent can keep running and keep reporting "done" while the records it saves no longer fit the customer's process. Nothing in the agent changed, so no release triggers a re-test. Treat changes in the customer's system as a release of their own: apply the change to a test copy, rerun the agent's riskiest tasks against it, and compare the saved records with the expected state before the change reaches production.

Teams that sell vertical agents into procurement, finance, legal or supply chain test before each release of the agent. The customer's ERP, procurement suite or CRM has its own release calendar, and its administrators change configuration between releases. This guide covers which of those changes need a re-test and how to run it.

Why does a system change break an agent that didn't change?

The agent's picture of the customer's system lives in its prompts, tool descriptions and field mappings. Those were correct on the day they were written. When the system changes, that picture goes stale without any error in the agent's own logs.

Princeton researchers made the same point in "Towards a Science of AI Agent Reliability" (February 2026, with results for 15 models released up to mid-2026, including Claude Opus 4.7, GPT-5.5 and Gemini 3.5 Flash). Benchmarks report one run in a fixed environment, while deployed agents meet migrated databases, changed API formats and updated tool libraries. The authors recommend re-running agents over time, because reliability can degrade as the environment drifts away from the one the agent was tested in.

The same paper adds a useful nuance. Format-only changes that kept the meaning (reordered fields, renamed parameters, other date formats) showed ceiling effects in that study. The risk sits in changes that alter meaning: a new rule, a new required step, a value that now means something else.

Which system changes should trigger a re-test?

Seven kinds of customer-system change that should trigger an agent re-test, from a new required field or status value to a changed role or permission.

Change in the customer's systemWhat can go wrongWhat to check after the re-test
A new required fieldWrites are rejected, and the agent retries or reports success anywayThe record exists and the new field holds a valid value
A renamed or new status valueThe agent's checks look for the old value and pass records they should stopRecords in the new status are handled like the old one
An extra approval or follow-up stepThe agent completes the write but skips the step the new policy requiresThe approval request or follow-up exists, in the right order
A changed validation rule or toleranceWrites that used to pass now fail, or values are rounded differentlyAmounts, quantities and units match the source
New reference data (cost centers, tax codes, units)The agent picks a retired code or a near matchThe code on the record is valid on the run date
A new API version or retired endpointCalls fail or return a different shapeOne correct write or a clean, reported failure
A changed role or permission for the agent's userWrites fail silently, or the agent can now reach records it shouldn'tAllowed writes succeed and disallowed ones stay impossible

The last row cuts both ways: a permission change can widen what the agent can do. The tests for that side are in AI agent permission testing.

What does a break look like?

An illustrative example. A procurement agent checks the supplier's status before it creates a purchase order and stops if the supplier is "On hold". The customer's admin team replaces "On hold" with two new values, "Suspended: quality" and "Suspended: payment". The agent's check still looks for "On hold", finds nothing, and orders from a suspended supplier. Its reply reads "PO created for supplier 4471", which is true. Every agent release test still passes, because the agent's own test copy still has the old values.

The same pattern appears in public benchmarks. EnterpriseOps-Gym (ServiceNow Research, March 2026) named two failure patterns that a system change can trigger:

  • Missing prerequisite lookup: the agent creates a record without first reading the data it depends on, so the record points at something invalid.
  • Cascading state propagation: the agent fails to trigger follow-up actions that a policy requires when a record changes state.

In that benchmark, policy-compliance checks had the lowest pass rates of the three verifier categories in the policy-heavy domains:

Domain (EnterpriseOps-Gym, Table 7)Task completionIntegrityPolicy compliance
IT service management53.7%50.6%30.0%
Customer service management58.3%70.0%45.5%
HR63.2%56.0%50.4%

A new approval rule or a new status is a policy change. These are the checks that a system change is likely to break.

How do you find out a change is coming?

You need two routes, because some changes are announced and some aren't:

  1. Announced changes. Ask each customer for their release calendar and for change notes from their administrators. Agree on this at onboarding: a planned change to objects the agent writes to triggers a re-test before it goes live.
  2. Unannounced changes. Run the agent's critical tasks on a schedule, nightly for example, against the customer's test copy. When saved records change and nothing changed on your side, the system or the model changed.

Model updates follow the same logic, including scheduled runs that catch silent changes; see model update broke your AI agent?.

How do you re-test after a system change?

Six-step flow for re-testing an AI agent after a customer system change, from updating the test copy to reading records back and deciding with the process owner.

  1. Put the change into the test copy first. Copy the configuration, not only data: required fields, status lists, approval rules, roles. See AI agent test environment: sandbox, seed data, reset.
  2. Rerun the riskiest tasks. Start with the release gate set: the writes with the highest cost to the customer if they went wrong. A short set of critical flows is described in smoke tests for AI agents.
  3. Add tasks that touch the changed object. If the status list changed, add a supplier in each new status. If a field became required, add a case where the source document lacks it.
  4. Read back and compare. Check the saved records field by field against the expected state, and check that nothing outside the task changed.
  5. Test the stop. Some changes make a task impossible for a while: a migration window, a deactivated user, a missing permission. In EnterpriseOps-Gym, a system in migration mode and inactive users were among the reasons a task was infeasible, and the best model refused infeasible tasks cleanly, with no side effects, 53.9% of the time. The expected result is a clear stop or a hand-off to a person, with nothing written.
  6. Decide with the process owner. A failure after a system change may need a fix in the agent, a change in the customer's configuration, or a temporary approval step. The person who owns the process decides which, as described in testing human-in-the-loop AI agents.

Who owns the re-test?

The agent vendor owns the tests and the agent's fix. The customer's admin team owns the change and the notice. The customer's process owner decides whether a failure holds the change or the agent. Write these three roles into the rollout plan for each customer, next to the list of objects the agent writes to. The write surface for ERP agents is covered in how to test AI agents before they get ERP write access, and the wider method in how to test vertical AI agents before release.

How does Agentiqa support re-tests after system changes?

Agentiqa is an AI Quality System for production AI agents. With Agentiqa, you describe each task in plain words. When you rerun the failed tasks, AI Quality Manager reports a step as a regression only if it fails again. Unconfirmed failures are listed separately.

First release gates are live within 7 minutes; for enterprise teams, an Agentiqa engineer sets up the gates on the riskiest tasks over a 3-day rollout.

FAQ

Why did my AI agent stop working after an ERP update?

The agent's prompts, tool descriptions and field mappings describe the system as it was. A new required field, a renamed status or an extra approval step can make its writes fail or land in the wrong state, even though the agent itself didn't change.

Which system changes need an agent re-test?

Any change to an object the agent reads or writes: required fields, status values, validation rules, approval steps, reference data, API versions and the agent user's roles. Format-only changes carry less risk than changes in meaning, but they still belong in a scheduled run.

How do you test an AI agent against a customer's system change before it goes live?

Apply the change to a test copy with the customer's configuration, rerun the agent's riskiest tasks and the tasks that touch the changed object, and compare the saved records with the expected state.

How do you catch system changes that customers don't announce?

On a schedule, for example nightly, against the customer's test copy. A change in saved records with no change on your side points to the system or the model.

Who decides what happens when a re-test fails?

The customer's process owner, with the agent vendor. The fix can be in the agent, in the customer's configuration, or a temporary approval step until both are aligned.

Sources

  • Rabanser, Kapoor, Kirgis, Liu, Utpala and Narayanan (Princeton University), "Towards a Science of AI Agent Reliability", February 2026; current version evaluates 15 models released up to mid-2026. arXiv:2602.16666. Recommendations on multi-run and multi-condition testing and on re-evaluation over time as databases, API formats and tool libraries change; environment and fault robustness showed ceiling effects for the format perturbations tested.
  • ServiceNow Research, "EnterpriseOps-Gym: Environments and Evaluations for Stateful Agentic Planning and Tool Use in Enterprise Settings", March 2026. arXiv:2603.13594. 1,150 tasks, 14 models; failure patterns "Missing Prerequisite Lookup" and "Cascading State Propagation"; Table 7 verifier pass rates by category; infeasible tasks (including system in migration mode and inactive users) refused cleanly at best 53.9%.

Next step

List the objects your agent writes to in each customer's system and ask each customer when those objects last changed. If you want to turn your riskiest tasks into release gates, we can go through them with you in 20 minutes.

Book a 20-Minute Vertical Agents Safety Review

Related posts