Debugging
Video freezes that looked like a software bug
Freezes, dropped frames, and audio that started early. The first assumption was a bug in the player. It was the operating system quietly moving the player's memory to disk.
What was happening
During ad transitions on a box, venues and testers saw four symptoms:
- Audio faded in before the picture changed.
- The last frame of one ad froze before the next one appeared.
- Frame rate dropped and playback stuttered.
- Under extreme conditions the screen stayed on the splash logo.
It looked like a timing bug in the player. Looking at the box instead of the code told a different story: it had about 7.6 GB of RAM and roughly 13.6 GB of memory committed across its containers, about 1.9 GB was sitting in swap, and the video daemon had taken around 12.8 million major page faults.
Making it happen on purpose
The field problem was intermittent. I could not fix what I could not repeat, so I used systemd cgroup limits to squeeze the video daemon's memory, added heavy disk I/O at the same time, and fired simulated ad tones.
| Setup | What happened | What it proved |
|---|---|---|
| Memory cap of 1 to 2 MB | Screen stuck on the splash logo. The pipeline waited forever for a reply. | The daemon must respond for rendering to move forward. |
| Memory cap of 20 MB | Transitions started 1.5 to 2.5 seconds late. | Less memory directly means slower transitions. |
| Memory cap of 6 MB plus disk I/O | The exact field bug: early audio, last-frame freeze, dropped frames, long delays. | Major page faults against flash storage are the cause. |
| Protected memory floor of 192 MB | Pages stayed in RAM and transitions were near-instant. | The fix works. |
Worst transition time
The fix, and what comes with it
The fix is a protected memory floor: a systemd override that sets MemoryMin=192M for the video daemon, so the kernel cannot evict its pages under pressure. I deployed it to live boxes and confirmed the results in the field.
The write-up also gave the team three follow-ups:
- Keep the override in the base provisioning scripts so every new display unit gets it.
- Track major page faults as a metric, so thrashing shows up before guests see it.
- Make sure the unified cgroup v2 memory hierarchy is active, because memory floors are only honored there.
Looking at one box also turned up separate problems on another: swap switched off and a disk at 94 percent. I restored swap and freed 21 GB.
The second case: sound that never arrived
On some boxes, every creative played in silence. The files were fine and each one had audio. Video was unaffected because it leaves the box by a different route.
The player runs as a root service. The audio system runs as a regular user on a private socket, and a private socket cannot be discovered. A process has to be told where it is. The shipped service file told the player where the display was, and nothing about audio, so it decoded sound and had nowhere to send it.
The fix is two lines of configuration. It had already been written weeks earlier on a branch and never merged. When I checked, the same two lines were missing from 13 of the 14 branches that carry the service file, including the one used for production. That is 93 percent of branches.
I validated the fix on hardware, wrote the ticket and a full write-up, and added a one-command check any engineer can run on a box. It prints 0 on an affected box and 1 on a healthy one.
The lesson I took from it: a ticket closed as "done" can still hide half a fix. The part that shipped solved the ad sync. The part that carried sound had no ticket left to live in.