Automation
The Music Mode test harness
A harness that treats the box as a system with a message bus, fires real tones at it, and checks every hop from trigger to broadcast. No one stands next to a TV with a stopwatch.
Why a harness
Testing Music Mode by hand means playing a tone, watching a TV, and guessing which component let you down. There are 32 tone slots, several audio sources, and switching that depends on timing. A person cannot cover that twice, let alone on every release.
So I built a harness that runs on the box itself, against the real message broker, and reports exactly where a chain broke.
How it works
Connect to the real broker
The harness joins the box's MQTT message bus, the same one the components use to talk to each other. It tests the product, not a mock of it.
Fire a tone
It sends a tone trigger for a chosen slot, the way the audio path would.
Tag and follow the event
Each tone gets a correlation ID. The harness follows that ID across the playback state change, content delivery, video start, video finish, and the return to broadcast.
Assert at every hop
Each step has a timeout (15 seconds per hop), and the whole switch from tone to confirmed ad has a 3-second budget. A slow pass is still reported.
Report with the chain
A failure names the step that did not happen, the expected value, the actual value, and the logs. It runs on demand and in CI.
slot 21 (custom length) tone fired corr=a1b2c3d4 playback_state state='ad' confirmed ad after 0.1s PASS content delivered duration=59s, audio on video/play ad_display video/finished within 120s PASS returned to broadcast yes PASS chain tone/trigger → playback_state → NextContent → CurrentContent → video/stopped → video/play → playback_state → video/stopped
What it tests
| Test type | What it does here |
|---|---|
| Functional | Mode entry and exit, and audio source switching. Music passes through without interruption while the mode is active. |
| Integration | Tone to controller to player to TV, across components, on live hardware. |
| Regression | Fixed bugs stay fixed. Known findings become cases that must keep passing. |
| Negative | Out-of-range tones (0, 33, 99, −1, 64, 1000) and malformed inputs must be rejected, not played. |
| Back-to-back trigger | A race-condition check. A second tone during playback must produce zero new video starts, and quick successive tones must not flip to music. |
| Timing and latency | The 3-second tone-to-ad budget and the 15-second per-hop timeout. |
| Persistence watch | A 10-minute watch that the state holds and does not drift back. |
| Soak | Long-running repeat runs to catch problems that only appear after hours. |
The harness also checks audio output and the external music source connected to the device, since a tone that switches the screen but kills the music is still a failure.
The six suites
| Suite | Tests |
|---|---|
| Standalone | 28 |
| Inventory | 42 |
| Persistence | 5 |
| Audio | 4 |
| Mode | 4 |
| Live, per slot | 7+ |
Before a run
The harness documents its own preconditions, because most false failures start there. The TV must not be in safe mode, the venue's music player must be plugged in and playing, tones must be uploaded in the venue app and on the music provider's site, and the harness files must be on the device.
What it found
The harness and the testing around it found more than 50 bugs before they reached venues. Examples:
- Back-to-back tones created switch-and-switch-back pauses.
- The wrong ad fired when a clip had been converted from .mov to .mp4.
- A deleted creative still triggered, because an orphaned slot file stayed on the box.
- The TV played creatives in place of the tone's clip.
- Music Mode jumped back to broadcast mode at random.
- Tones behaved differently on two box generations (V1.0 and V1.5).
Scope discipline. New requirements (mode exclusivity, dynamic-length tones, legacy slot migration) arrived after I started. I kept the harness story at its committed scope and tracked the extra coverage in its own ticket, so the first version shipped on time.