Testing AI-generated code takes four layers: static gates (types, lint, security scanning) on every change; the agent's own unit tests, which you keep but audit instead of trust; integration tests that hit real dependencies rather than mocks; and independent verification, where something that did not write the code checks what the running system actually does. Most teams stop after the second layer. The failures that reach production live almost entirely in the third and fourth, and when the code is an AI agent that writes to customer systems, the fourth layer has to become a release gate.
Why the agent's own unit tests aren't verification
An agent that misreads a requirement writes code implementing the misreading and tests asserting it. Coverage rises, the checks go green, and nothing has checked the work against reality. CodeRabbit's study of 50,000 pull requests found AI-written pull requests carry roughly 1.7x more issues than human-written ones, while other research shows AI code passes more automated tests than human code. More defects and more green checkmarks, from the same code.
The failure modes are consistent. Generated tests mock anything inconvenient, so the seams between components go unexercised. Assertions drift toward tautology, checking that a mock returned what the mock was configured to return; Autonoma documented this pass-but-don't-assert pattern. Our review of AI test case generation reached the same verdict: useful scaffolding, weak judgment.
None of this makes AI-assisted development a bad trade. DORA's 2025 report found about 90% of developers now use AI, with higher throughput alongside more delivery instability. Output scaled; verification didn't.
Doesn't the agent already check its work in a browser?
Partially. Coding agents can open a page, take a screenshot and report success. That proves the page renders, not that the flow works: a screenshot of a login form says nothing about whether valid credentials reach the dashboard, or whether an order still saves correctly after a refactor two files away.
The author problem applies here too. An agent grading its own work brings the same assumptions it coded with, so its check inherits its blind spots. AI-generated code increases the need for testing rather than reducing it, because volume goes up while author-level assurance goes down.
How to test AI-generated code: four layers
Layer 1: static gates that don't negotiate
Type checks, linting, dependency and secret scanning, plus whatever security analysis your stack supports. They are cheap and catch a real class of agent mistakes: hallucinated APIs, unsafe patterns, dependency confusion. Make them hard merge blocks. Know the ceiling, too: they miss architectural drift, which Martin Fowler's essay on sensors for coding agents covers well.
Layer 2: keep the agent's tests, but audit them
Generated tests are worth keeping as regression scaffolding once a human has read them. For each one, ask whether the assertion could ever fail, whether it checks behavior or restates a mock, and whether the fixtures resemble production data. Delete the tautologies and regenerate with real fixtures.
Layer 3: integration tests against real dependencies
Run the modules the agent touched against a real database, real queues and real internal services, or use contract tests where a live dependency is impractical. This layer exists to puncture the mock-everything pattern. Integration tests are slower and flakier, which is exactly why agents avoid writing them, and why they find what unit tests can't.
Layer 4: independent verification of outcomes
Independent means a different author: a person or tool other than the agent that wrote the code exercises the running system through complete flows and checks the result. Manual click-through works but doesn't scale. Scripted end-to-end suites (Playwright, Cypress) give deterministic checks, at the cost of maintaining selectors for a UI the agent may regenerate weekly. Agentic testing uses an AI agent to explore the app and test flows like a user, without pre-written scripts.
Whichever you choose, check outcomes, not appearances. "The page loaded" is not verification. "The record was saved with the right values" is.
When the code is an AI agent with write access
The stakes change when the AI-generated code is itself an agent that creates purchase orders, updates CRM records or posts to a ledger or billing system. A prompt change, a model update or a slow upstream API can break one of those writes without failing a single unit test, because the tests run against mocks while the damage happens in your customer's system. Typical failures:
- Corrupted IDs and fields: an ID truncated on write, or two parameters swapped.
- Unit and amount mix-ups: a line booked in pieces when it should be meters.
- Retry storms: a 502 or timeout makes the agent repeat the same write and create duplicates.
- Hallucinated success: the agent reports "order created" when the step never ran.
For these agents, layer 4 has to become a release gate. Before a release reaches production, the gate runs the agent's critical write flows, including injected 502s and timeouts, and checks what was actually written: the right record, the right IDs, amounts and units, exactly one write, and a saved state that matches what the agent reported. Green means the release can go. Red stops it, with the execution trace, the data diff and the steps to reproduce. A fuller definition is in what is a quality gate for AI agents?
This is what Agentiqa builds: high-confidence release gates for vertical AI agents in high-liability domains such as procurement, legal, finance and supply chain. We wrap your agent's critical ERP, CRM and billing write flows in release gates, and we keep them current as prompts and models change. First release gates live within 7 minutes. The pattern behind it is covered in how to test AI agents that write to your CRM or ERP and how to catch hallucinated success. (If you're an individual developer who wants to point an AI tester at your own web app or a simple agentic workflow, start free with the Agentiqa Community Edition: 20 runs a month, reset monthly. Pro is $39/month when you need more.)
What should change on your next AI-heavy release?
- Make types, lint and security scanning hard merge blocks.
- Read the generated tests. Keep only the assertions that could actually fail.
- Run integration tests against real dependencies for every module the agent touched.
- Before each release, verify your highest-value flows end to end, by a human or an independent tool, never the authoring agent.
- If your product's agent writes to customer systems, gate releases on those writes, including failure cases such as 502s and timeouts.
- Track escaped defects and time-to-verify instead of coverage.
FAQ
Can a second AI model review the code instead of testing it?
Cross-model review is worth doing, since a different model catches some of the authoring model's mistakes. But review reads code; it doesn't run it. Integration failures, broken flows and bad writes only appear when the system executes. Treat second-model review as an extra filter alongside layers 1 and 2, not a substitute for layers 3 and 4.
When should each layer run?
Layers 1 to 3 run on every change; they are fast enough to run before code is merged. Full-flow verification usually runs on a release candidate before production, because it takes minutes rather than seconds. For an agent with write access, that pre-production run is the release gate: a red result stops the release.
How is a release gate different from observability tools like Langfuse, Arize or Braintrust?
Observability tools show what your agent did after it ran in production. A release gate runs before the release ships and stops it when a check fails, so a bad write is caught before it reaches your customer's system. Many teams use both. We compare them in release gates vs observability and evals.
Who owns a bug in code no human wrote?
The engineer who prompted and merged it. Agents don't attend incident reviews. That is why verification must be independent while accountability stays human: someone has to read the failure, judge severity and own the fix.
Where to start
Pick the three flows that would cost you the most if they broke and verify them independently before the next release. If those flows are an AI agent writing into customer ERP, CRM or billing systems, start with the writes. In 20 minutes we map your agent's write surfaces and show you a release gate stopping a live failure.