What AI changes in testing, and what it does not

You tend to hear two versions of this. One says AI changes nothing in testing, that a model producing plausible test cases is a party trick. The other says the tester is finished and QA is now a software licence. Neither matches what we see on projects. The split that actually helps is between the parts of testing that repeat and the parts where somebody has to decide something.

The repetitive part is where most of the hours go. Drafting a hundred test cases from a requirements document. Rewriting a defect so a developer can act on it. Comparing this cycle against the last one. Spotting that nothing has touched a whole area of the product. It is patterned work, and doing it well is largely a matter of doing it consistently.

The other part is deciding things. That a flow matters more than the requirements suggest, because of how customers actually use it. That an unremarkable-looking defect is the one that will force a rollback. That a release is not ready, on a date that has already gone out. None of that repeats in any useful way. It comes from knowing the product, and from having been wrong about something similar before.

AI is good at the first. It does not do the second at all.

• First-draft test cases from a requirement or a user story, which a tester then edits, cuts and reorders
• Defect triage: grouping duplicates, proposing severity, tidying reproduction steps into something a developer can act on
• Coverage analysis: pointing at the areas of a product where nothing has been exercised
• Maintenance of automated suites, where a change in the application breaks a batch of scripts in the same way
• Summarising a long cycle into something a stakeholder will actually read

• The release decision. Ready, ready with known risks, or not yet: that carries consequences, and a person has to sign it
• Deciding what matters most in a product, which comes from the business and its stakeholders, not from a model reading a backlog
• Exploratory testing, where the value is precisely in following a hunch that no specification describes
• Anything that would require sending your production data or your source code somewhere it is not supposed to go

Asking a supplier whether they use AI stopped being informative around the time everyone started saying yes. Narrower questions work better. Which steps does it touch? What does a person check before anything reaches you? Where does your data go? Who signs off that a release is ready, and on what basis?

A supplier who can answer those is supervising the tool. One who cannot is either not using AI at all or not watching what it does, and you would want to know which before you sign anything.

For our part, we use it on case generation, triage and coverage analysis, where it cuts the upkeep. A person decides whether something is ready to ship, every time, and can explain that decision without pointing at a model.