AI moves testing from brittle scripts to durable intent and evidence.
A BOOK BY JASON ARBON
Test at the speed of AI.Software is easy to generate.
Confidence is hard.
How AI Tests Software is a field guide to using AI to inspect, explore, test, challenge, and qualify the software that people and AI agents build.
AI CHECKS•AI TESTS•AI CONFIDENCE
THE QUALITY PROBLEM HAS BECOME THE CONTROL POINT OF SOFTWARE.
01 / THE THESIS
AI is subsuming
the old testing stack.
Test frameworks, recorded scripts, test-case databases, issue trackers, and reporting dashboards were abstractions built so people could manage software complexity. AI can work directly from intent, source, screenshots, logs, traces, and the current product state.
The reusable asset is no longer a fragile click sequence. It is the promise, the evidence contract, the authority boundary, and the decision the result can change.
The coding agent can orchestrate validation, but independent validation has to qualify the result.
AI will spend more compute validating, comparing, repairing, and monitoring software than initially generating it.
02 / A BETTER VOCABULARY
Four terms for a new quality system.
Stop using “AI testing” as one vague bucket. Name the capability, the interaction, the workflow, and the decision.
01
AI Checks
Use screenshots, logs, network activity, DOM, traces, and source to inspect quality evidence without touching product state.
02
AI Tests
Drive browsers, devices, APIs, data, and the whole system to prove a user or business intent.
03
AI Test Harnesses
Run the long-lived workflow: context, planning, execution, challenge, reporting, and AI Confidence.
04
AI Confidence
Decide whether the evidence is enough to ship, canary, hold, repair, or gather more proof.
03 / THE LOOP
Generate. Validate.
Learn. Repeat.
An AI Test Harness turns a fresh repository, a change, a URL, or an existing suite into a stateful quality workflow. It can discover risk, generate candidates, execute them through any adapter, challenge its own findings, and continue until AI Confidence converges.
LONG-RUNNING QA WORKFLOW
- 01Context + riskWhat changed? What matters?
- 02ExecuteRun the right checks and tests.
- 03ChallengeTry to disprove the findings.
RESULTAI ConfidenceShip, hold, or gather more evidence.
04 / INDEPENDENCE
The creator can ask
for the test. It cannot
be the only judge.
No one buys a house without an independent inspector. No one accepts a bridge because the contractor says the concrete looks good. AI-generated software needs the same separation.
The coding agent orchestrates from the richest possible context. An independent cloud validation plane gathers its own evidence, challenges the result, and qualifies what is ready to ship.
START HERE
Test at the speed of AI.
Qualify what ships.
Join the early-access list for book updates and new material.