Testing Manuals

Testing ha become a native part of any system development. Especially for software systems (what is not a software system today, after all?). And the common sense is that there is always not enough of testing and that most systems ship with bugs that surface while the system is in use, wrecking havoc or causing minor inconveniences.

One, often forgotten part of testing is user manuals and procedures. Test systems are built by engineering teams. Manual tests are run by engineers. And the engineers usually "know" how the system under test is expected to perform. Missing the gap that may exist between a system and its manual.

This happened to us last week, during a fairly routine procedure of testing the integration of our software into customer's product. The product was line powered (230V~) and simply exploded after power up. We found later there was a bug on the label, where terminals for N (Neutral) and L (Live) were shifted so both L and N led to the ground plane, resulting in a short circuit. Obviously the customer tested this product before shipping to us, but very likely they tested it either before sticking the label or without looking at the label, as they simply "knew" how to connect it. In the end it was not a big deal, the label will be fixed and the product is good to go.

But this same mistake might have been a contributing cause to the death of almost 200 people on the fatal Lion Air crash, where pilots were fighting with a misbehaving flight control system. It is hard to imagine that this system had not been tested thoroughly. The aviation industry is highly regulated and test procedures there are probably the most strict. But the question is, was this tested to behave properly as designed, or as documented in the manual? Because the tragic thing is (setting the faulty sensor aside) the bug was not in the software, it was in the manual.

The bottom line here is: a bug in a manual may have the consequences as severe as a bug in a system. And I highly doubt manuals, in general, are covered by as thorough test procedures as systems are.