Design or defect: how to brief an external QA partner so your tracker stays useful

Two simple cards side by side, one labelled as a defect and one as a design question

A first external QA pass can fail quietly. The tracker fills with notes such as "the jump feels floaty" or "this enemy is too hard." A note like that is not specified enough to classify from the wording alone. It might be an intentional feel choice, or it might be a regression, so it needs clarification before anyone assigns it to the design or defect lane. When unclear notes pile up in the tracker instead, crash and progression bugs sit behind them, and trust in the partner starts to slip.

The useful move is a short brief, agreed before the pass starts. It should tell the partner what expected behaviour looks like, where a design question should go, and how a defect should be written. The rest of this guide shows how to set that brief, and how to judge the first week against it.

Write the expected behaviour for the journeys in scope

For each player journey the partner is covering, state what "working" looks like in plain language: the first session completed, a save restored, a purchase granted, a match joined. This is not a design bible, and it is not a full regression catalogue. It is the expected-result line the tester will compare against.

If a journey already has an oddity the studio accepts, say so in the brief, so it is not logged as a new defect. A ranked handful of journeys with clear expected results is more useful than a long list with none.

Give design questions their own lane

Agree a place for design questions that is not the defect pile, or a clearly labelled ticket type that producers already watch. Name the person who answers those questions. Say how quickly a blocked tester can expect a reply.

A design question that sits in the same pile as a crash, and carries a severity as if it were a crash, is how a tracker goes stale. The partner should still raise the question. It just should not look like a blocker unless the studio has said that it is one.

Keep defect tickets for broken behaviour

Keep defect tickets for broken behaviour. If the partner can point to an expected result the studio briefed, and the build does not meet it, that belongs in the defect lane. An undocumented regression, or other objectively broken behaviour, still belongs there too, even when the brief never named that result. A missing expected-result sentence is an uncertainty to resolve, not a reason to move an observed failure into the design or taste lane on its own.

What a design-question note should contain

A useful design-question note still names the journey, why the behaviour might be intended, and what happens to the player if it is wrong. It should not carry a crash-level severity on an underspecified feel note, and it should not demand that design change in this pass.

Once an item is marked as intended or deferred

When the studio marks something as intended, that decision should be visible to the next run, for example on the known-issues list the studio already keeps, so the partner does not log the same item again unless the behaviour has changed.

A decision that an item will not be fixed in this pass is not the same thing. That item can remain a known defect even if work on it is deferred. Keep it on the known-issues list, still as a defect, with the agreed rule for when it should be retested or escalated, including when the impact or the scope changes. Do not treat deferral as a finding that the behaviour is intended, and do not open a duplicate ticket on the next build unless that rule says to raise it again.

How to judge the first week

Do not treat the ticket count as a score, and do not look for a "good ratio" of design notes to defects. There is no such figure to aim at. A fair test is whether the defect tickets are clear enough for the studio to triage and act, whether design questions reached the named person, and whether the same underspecified note appeared twice. That tests the partner, and it also tests the brief the studio sent.

Engineers at the studio own the fixes. The studio also owns the release decision. A clean first week is evidence that the working relationship is sound. It is not a store approval, a certification, or a promise about how the title will do once it ships.

Ready to brief the first pass?

GameCloud's founding year is FY 2010-11. PC, iOS, Android, TV, Web, XR and other emerging platforms are the targets we test for studios. Email Sales@GameCloud-Ltd.com a short note of the journeys in scope and how you want design questions raised, and you will receive a written scope and estimate.