Power Cycle Kills

Last week I had an overnight layover at the San Francisco airport. Landed in the afternoon and then took the hotel shuttle bus to the nearby hotel. It turned out the bus could not enter the hotel parking area: the automatic barriers were lowered and a number of cars before us was already lined up. We ended up disembarking the bus on the street and walking up to the hotel. The front door did not open.

"We have a power outage" - a staff member explained - please use the side door. I did. Then the check-in was by means of writing my name and CC number on a piece of paper and walking up the stairs to the 3rd floor. I still could open my room with the keycard (battery-operated?) but inside it was dead silent - no usual ventilation hum. There was no WiFi either. 

They fixed the power in the evening, but many systems still did not work. Elevators required a service company next day to get started. WiFi was on but the captive portal did not work until next morning.

This whole situation reminded me of the research on some competing wireless lighting control systems I did some time ago. It turns out the topology-aware networks (like ZigBee and Thread) have fundamental issues with network formation.

A topology-aware network must be formed before it carries traffic. Nodes exchange link establishment messages at startup and periodically, negotiate parent-child relationships, elect routers, and exchange frame counters with neighbors to establish replay protection. This is not a one-time commissioning step — it happens again after every power event.

Simulation from TU Delft, conducted with Philips Lighting Research: a 100-node network leaves 9 nodes unattached; a 256-node network leaves 78 unattached — 30% — with the router count settling at 23 rather than the 32 the specification permits. Failures are attributed to congestion, interference, and node density exceeding stack limits. Separately, 88% average availability was measured on a very small 38-node network.

It's silent and partial. The failure mode isn't "network down," it's "some nodes never joined." No alarm fires. A commissioner sees fixtures that don't respond and cannot distinguish an unjoined node from a wiring fault, a dead driver, or a device out of range — so diagnosis is by elimination, on a ladder.

It repeats. A network that formed successfully may fail to form again after a power cycle. Every outage, every breaker test, every construction-phase de-energization is a fresh roll of the dice. Failure produces artifacts, not just absence. A unit that cannot join is specified to form its own network. So the result of partial formation is a scatter of spurious single-node networks that must be found and re-commissioned.

Recovery competes with the cause. Topology maintenance already dominates traffic — roughly 47% are link establishment messages on a measured network. Under congestion, more nodes lose their parents, generating more attach traffic, in the same congested channel. Reported recovery from a short traffic burst is around ten minutes. One marginal node can prevent convergence entirely. A node operating at the edge of radio range attaches and detaches continuously, forcing perpetual topology rebuild. With several such nodes the network can be saturated by management traffic and never converge, leaving little capacity for the application.

The contrast:

Bluetooth Mesh / NLC has no formation phase. It requires no topology, so there is nothing to converge on and nothing to rebuild. Provisioning is per-device and independent: a node is usable the moment it is provisioned, the outcome is deterministic and observable per device, and a power cycle restores operation without any network-wide negotiation.

I wrote on this in the past blog post: https://headworx.slupik.com/2022/05/thread-when-healing-is-breaking.html. But it is worth repeating. 

The reframe: in a topology-aware network, "commissioned" and "operational" are different states separated by a probabilistic process that runs again every time the power blinks. That gap is where return service visits come from.