Testing (Software)

Testing the software turns out to be the major cost of product development for us. It is mostly responsible for moving a typical feature from a PoC (Proof of Concept) state to a fully production quality state. Depending on a feature, the PoC is 20% to 10% of the effort. And the PoC works (that is the whole purpose of a PoC). So from the management point of view it is sometimes hard to swallow - the fact that we need to work 5x (or 10x) longer to release the feature. But it is all much more complex in embedded, where you fight for every piece of a resource, such as Flash, RAM, system interrupts, hardware timers etc.

Having said that, testing is really one of the outcome of the software revolution. The fact that you actually CAN test. Test repeatedly, test every build, test, test, test... In our infrastructure we run several thousand automated tests on a typical build of what is considered "a firmware for a lightbulb". The tests take a whole night. And the multi-year statistics tell us the final product is essentially bug-free. The products "just work" and in the past we had to release a patch only twice (and the patched bugs were never affecting majority of users, addressing some corner cases instead).

Now imagine a world in which you simply cannot test. How would product development look like. I know it may sound abstract but it actually is not. There is a great story on YouTube on the American Mark 14 Torpedo. It was a new weapon, which turned very expensive, and almost impossible, to test. The outcome was a weapon that was failing most of the time. And historians playing what-if scenarios have even been predicting a different outcome of the WWII, should the Mark 14 actually have worked...