Process

Building QA from zero

I joined as the first QA engineer. There was no shared definition of "tested", so release readiness came down to judgment calls. My job was to replace that with evidence anyone could read.

Test plansTest-record templatesRun logsRunbooksRelease criteria

The problem

Problems showed up in the field, in front of guests, and reports did not always carry what a developer needs: the box generation, the software version, the steps, and the evidence. Fixing a bug starts with reproducing it, and that needs those details.

The product is also hard to test casually. One feature touches a web app, a cloud backend, a message bus, a player on the box, a TV, and sometimes a music player plugged into the box. A "pass" on one layer means little if the next layer is broken.

What I put in place

Per release

Validation scope and success criteria

For each release I defined what was in scope, what had to be re-tested (regression coverage), the edge cases worth the time, and feature-specific success criteria. That list is what "ready" means.

Per test

The test-record template

One page per test, with the same fields every time, so any engineer can pick up a record and rerun it without asking me anything.

Per harness

A documentation set

Every harness ships with an operational guide, a case catalogue, a runbook, a test-record template, and a support document. They all follow one shared harness standard.

Inside a test record

Section	What it captures	Why it matters
Header	Title, section, date, type (functional and so on), stop-point flag	Tells a reader what this test is and whether a failure should halt the release.
Environment	Box generation, where it was set up, backend project, venue app build, music source, screen under test, container tags	Most "flaky" results come from an environment nobody wrote down.
Preconditions	For example: TV not in safe mode, music player plugged in and playing, tones uploaded, harness files on the device	Separates "the feature failed" from "the setup was wrong".
Steps and expected result	The exact action and what should happen	The expected result is written before the run, never after.
Run log	Run number, date, tester, what changed from the baseline, result, what happened, evidence	History of every attempt, including the failures.
Findings and logs	What the run taught us, with raw logs attached	The part developers actually use.

A real run record

This is the shape of an actual early record. I ran four tone slots through the harness on a staging setup.

run #1 · result: FAIL9 min 57 s

expected   slots 21, 22, 6, 24 each play their ad, then return to broadcast
slot 21    ad played, finish never seen within 120 s
slot 22    ad played, finish never seen within 120 s
slot  6    ad played, finish never seen within 50 s
slot 24    nothing switched (slot empty)
mid-playback second tone  ignored as designed   PASS
summary    4 failed, 1 passed

The finding that mattered. Switching between tones back to back could make music play instead of the ad. That is exactly the failure you do not want to meet in front of a customer, so it went into the demo notes the same day.

Why the documentation follows the tests

A test plan forces one habit: write down what the software should do before you check what it does. Because every behavior had to be stated as an expected result, my test plans ended up as the source for the software documentation. The team reads the same pages I test from.

The effect on developers was practical. Bugs arrived with an environment, steps, and evidence, so they were reproducible without a call.

Tools: Google Docs templates, Confluence runbooks and guides, Jira for tracking.
Scope control: when new requirements arrived mid-work, I kept the original story at its committed size and tracked the added coverage in a separate ticket, so estimates stayed honest.