The shift to AI-assisted testing

Developers are using AI to write code. Product teams are using it to turn ideas into requirements. Testers are using it to generate scenarios, analyse defects and keep automation alive. The opportunity is real enough: it takes repetitive work off a small team and lets them cover more ground.

The catch is that producing more testing activity and achieving better quality are not the same thing, and it is easy to buy the first while believing you bought the second. Software can now be generated faster than teams can safely understand it, which puts more weight on the discipline rather than less.

AI-assisted testing means using AI to support a QA process you already have: generating test ideas, drafting automation, summarising logs, grouping defects, producing synthetic data.

Testing AI systems means testing a product that itself runs on machine learning or generative AI. That needs extra techniques, because the outputs are probabilistic and shift with the data and the context. Hallucination, grounding, prompt injection, data leakage and consistency across repeated runs all become things you have to test for.

The distinction matters because the controls are different. A test case drafted by AI still only needs the review any test case gets. Testing an AI assistant needs considerably more than that.

For the first shift, the split of responsibility is the useful thing to be clear about:

Start with one repetitive task you already understand well: drafting scenarios from stable requirements, finding gaps in a regression pack, summarising logs. Work like that is easy to review and easy to undo. An autonomous agent with permission to change environments is neither.

1. What problem does it solve, and how will we measure the improvement?
2. What data does it receive, where is that data processed, and who can access it?
3. Can we review, reproduce and audit its output?
4. What happens when it is wrong, unavailable, or confidently misleading?
5. What permissions does it have, and can they be limited?

For any tool, five questions are usually enough to separate a product from a demonstration:

AI tools want requirements, source code, logs, test data, screenshots and tickets. Buried in that material are credentials, personal data, commercial terms and the shape of a client’s systems. Agree a usage policy before the first pilot, covering which tools are approved, what data must never go near them, who owns the accounts, how long anything is retained, and how someone reports a suspected exposure.

Then apply least privilege. An assistant that drafts a test has no business changing test data, editing a repository or deploying anything. Start read-only. Work in test environments on synthetic or masked data. Never let an agent near production or a third party’s system without written authorisation.

Where the product under test is itself AI-enabled, the OWASP GenAI top ten is a workable checklist to start from: prompt injection, sensitive-information disclosure, supply-chain and model poisoning, improper output handling, excessive agency.

"Human in the loop" is not a control unless that human has the time to look properly, the information to judge, and the standing to disagree without it costing them. A reviewer approving hundreds of generated test cases without reading them is a rubber stamp, and everyone in the room usually knows it.

Build the oversight around decision points instead. Low-risk output such as draft ideas, summaries and classifications can be sampled rather than read line by line. Anything that identifies a release-blocking risk needs an experienced person making the call, with the evidence kept. Anything touching production, customers or a third party needs written authority and explicit approval, and no autonomous action whatsoever.

A QA team should be able to say without hesitating who approves an AI-generated result, who can override it, and at what point the work stops. One more thing worth doing: where you can, keep the system that generates the output separate from the one that evaluates it.

The credible business case was never that AI replaces the team. It is that a skilled team spends less of its week on repetitive work and more of it on risk and exploration.

Those gains are real, and they are not automatic. DORA’s research on AI-assisted development discusses managing a productivity dip during adoption. A randomised METR study of 16 experienced open-source developers working on 246 real issues found that allowing AI made tasks take 19% longer on average in that setting. One study in one configuration does not prove AI slows every team down. It is a decent reason to measure your own workflow rather than trust a vendor’s claim, or your own impression of how fast things feel.

So measure the whole loop rather than the generation step. Time to design, execute, analyse and maintain. Escaped defects, false positives and the corrections made at review. Coverage of the journeys that matter. Cost with review time and rework counted in. If AI saves thirty minutes of drafting and creates an hour of review, it has cost you half an hour.

The teams that come out of this well will not be the ones that bought the most tools. They will be the ones that knew their risks, trained their people, kept their data where it belonged, checked AI output against evidence, and kept human judgement at the points where it counted.

Use it to cut repetitive work and to surface patterns that are hard to spot by hand. Do not let speed stand in for understanding, and do not mistake an impressive output for a reliable one. The point is to extend what the team can reach, not to hand off responsibility for quality.

• NIST AI Risk Management Framework — https://www.nist.gov/itl/ai-risk-management-framework
• OWASP GenAI Security Project, Top 10 for LLM and GenAI applications — https://genai.owasp.org/llm-top-10/
• ISTQB Certified Tester AI Testing (CT-AI) — https://istqb.org/certifications/certified-tester-ai-testing-ct-ai/
• METR, measuring the impact of early-2025 AI on experienced open-source developer productivity — https://metr.org/blog/2025-07-10-early-2025-ai-experienced-os-dev-study/
• DORA, the ROI of AI-assisted software development — https://dora.dev/ai/