Traditional QA was built on a contract: the same input yields the same output. That contract is why a screenshot diff, an assertion, and a recorded fixture still work. It is also why so many AI programmes are currently arguing with their own tests.

A language model is allowed to be different and still be right. A retriever is allowed to rank sources in a new order after a re-index. An agent is allowed to take another path to the same goal. Treat that as flake and you will either freeze the product so it cannot improve, or mute the suite because it cries wolf.

Keep the boolean where it still earns it

Not everything became probabilistic. Parsers, schemas, policy checks, and tool adapters still have a correct answer. Those parts should stay exact. The mistake is asking the model path to behave as if it were a parser.

• Unit the deterministic edges: contracts, permissions, adapters, fallbacks.
• Evaluate the model path against a golden set with pass bands, not single strings.
• Replay production-shaped traffic in staging, and watch the score move when you change a prompt.

“

The teams that ship AI confidently are not the ones with the most tests. They are the ones with the clearest definition of good enough — and a way to measure it every time the system moves.

Pass becomes a distribution: relevance, factuality against a source, safety boundaries, tone within a band, tool-call correctness, regression against a known baseline. You still write tests. You change what they are allowed to prove.

We build that stack as quality engineering, not as a science project parked beside the pipeline. If your current suite only knows how to say equal or not equal, it is time to change the question it asks. Request a call if you want help writing the new one.