Evidence every cycle, not a report at the end

Ask what a testing engagement produced and you will often be handed a percentage. Ninety-four per cent of cases passed. It is a comforting number that answers almost nothing. Which six per cent failed? Were those the cases that mattered? What does any of it mean for a customer?

A release decision comes down to one of three statements. This is ready. This is ready with these known risks. This is not ready yet. Whichever one you pick has to hold up in front of somebody who was not in the room, whether that is a stakeholder, an auditor, or whoever ends up explaining the outage.

• Per case: what was executed, in which environment, against which build, and what actually happened
• Per defect: severity, reproduction steps, environment, and the business consequence in plain language
• What was not tested, and why: excluded scope belongs in the record as much as covered scope does
• Retest results, so a fix that was claimed can be shown to hold
• Open risks at the point of the decision, stated as risks rather than buried in a summary

Evidence that arrives at the end is a record of what happened. Evidence that arrives during the cycle is something a client can still act on. That is the whole difference.

It changes the relationship too. When findings arrive continuously, a client can tell within the first week whether the testing is chasing the right things, and redirect it if not. When the first substantial output is a document in the final week, the only thing left to do is accept it. Reporting as you go is what makes a QA supplier auditable while there is still time for that to be useful.

There is a second effect, and in practice it may be the more important one. When the evidence is already visible, a finding is much harder to quietly soften as the date gets closer. Anyone applying pressure has to argue with the record rather than around it.

• Findings visible during the cycle, in your tracker, not a summary at the end
• A written statement of what is in and out of scope before anyone starts
• Defects raised with business impact, not only technical reproduction
• A named person who will say ready, ready with risks, or not yet, and stand behind it
• The suites, runbooks and evidence staying with your team when the engagement closes