Testing is Not Enough

Tuesday, November 13, 2018

#Functional Safety    #Automotive SPICE

The Automotive industry relies heavily on testing and “proven in use” to help ensure the quality and safety of their products.  Because we do so much testing, the software has to be good, right?  Unfortunately there are numerous examples where testing hasn’t caught all the problems, and consumers have paid the price.

OEMs or other customers aren’t usually concerned with defects unless they cause problems, increase their costs, or delay their work. Defects, on the other hand are very important to product developers. Defects force them to rework defective features instead of developing new or improved features. Defects are important to management because they increase development costs and can delay schedules by weeks, sometimes months as developers fix defects in test. Often organizations:

  • Spend more than 50% of the schedule in test;
  • Devote more than half of their resources to finding and fixing defects;
  • Dannot predict when they will finish; and
  • Deliver poor-quality and over-cost products.

A 50,000 LOC system with traditional development and testing methods would:

  • Have 25+ defects/KLOC at test entry [1250 defects],
  • Take 12,500+ hours* to find and fix [6 years of work], and
  • Be late and over budget.

* The industry average is about 10 hours to find and fix each defect.

What do the numbers look like in your organization?  Check the most current project schedule and resources, and post your findings in the comments.

Software is the only modern technology that relies primarily on testing to ensure quality even though there is a tremendous volume of data that shows removing defects before test is faster, cheaper, and more efficient than removing them in test. Don’t get me wrong, testing is essential, but it must be used properly because it is inefficient. Testing should verify that all defects have been removed and validate the requirements have been implemented properly. Testing becomes less effective when there are many defects. Defects often hide other defects, defects are created when we fix defects, and repairing simple defects takes time that could be used to test for more subtle defects.

Because of the complexity of automotive products, it is virtually impossible to test every combination and permutation. So a risk based strategy is employed. We approach testing like clearing a minefield. The paths you test are usually free of mines, but you pay a price when you stray off the cleared path. Testing by itself is not enough, it is rarely more than 50% effective! Think about that, if you find 100 defects in System Test then there are still at least 100 out there for someone else to find. Hopefully those mines are duds.

If just testing isn’t enough then what should you do? The ASPICE models and the Functional Safety standard [ISO 26262] provide pretty clear guidelines and requirements for verification and validation techniques that can and must be used to help build a quality and safe product.

But even fully applying the model and standard are not enough! To get a quality and safe product out of test, you must put a safe and quality product into test! In order to achieve that you must plan for Safety and Quality from the beginning of the project.

Look for future posts regarding the economics of testing and why you must plan and manage quality, and simple ways to review the effectiveness of your verification and validation methods within your organization.  A number of our clients have seen significant reductions in cost and schedule with improved quality. Contact us to find out how you can incorporate these best practices.


Do you need to improve your automotive product development, to increase efficiency, or to comply with ASPICE and Functional Safety? You are at the right place.