How Do I Test This Requirement?

“All I want for Christmas are testable requirements” sings the test engineer.  “Jingle bells, jingle bells QA missed a test” mocks the system architect.  At the holiday party, the developers and the testers vow to work closer next year to prevent escapes even though everyone knows that this system is hard to test.  Sound familiar?  Automotive SPICE has a concept called Verification Criteria that can be used effectively to bridge the knowledge gap of test practices between the system engineers or architects and independent test teams.  In this post we will clarify what verification criteria are, how to effectively use them, and how they support ISO 26262 implementations.

Verification Criteria are referenced in the ASPICE PAM 3.0 model at the system level (SYS.2 BP5 and SYS.5 BP2) and at the software level (SWE.1 BP5, and SWE.6 BP2).  They define how to test a requirement (the standard calls these qualitative measures) and any specific parameters that must be used in the test cases (the standard calls these quantitative measures).  The verification criteria are not the test plan or the test cases; they are important details about the testability and the required measures to verify a requirement.  They are ‘hints’ to the test plan developer to ensure that a requirement is properly tested.  They are like requirement presents given by the system or software engineer to the tester.

Verification criteria can be used to increase test coverage, ensure that a requirement is testable, and spread the goodwill of knowledge sharing among the project team.  In order to accomplish all of these good things, the criteria need to be easily accessible and linked to the requirement.  This is typically done by adding a text attribute to the requirements record in Doors, Jama, etc.   It is also best practice to add a check for verification criteria in the requirements review checklist in order to ensure each requirement has proper verification criteria.  Other ways to do this is to add them into agile stories, model attributes, or into a test case management system.  The point is to link them to the requirement they support so that they are easy to find.

Agile stories ought to include, among other details, acceptance criteria.  In some cases, the acceptance criteria are the tests themselves.  However in the automotive industry, where the test cases require traceability and are often stored in a test case database, the verification criteria could be included in the story card as acceptance criteria within the Definition of Done (For a good discussion on DoD, please see Peter Abowd’s post Agile and Safe).  This strategy would support just-in-time test case creation (creating the tests and verifying the feature all within the current iteration) by providing the test team with important testability criteria.

The ISO 26262 standard for functional safety requires various methods for developing test cases.  Verification criteria could be used to guide the tester with criteria for applying those methods to the particular system.  For example, ASIL C and D safety requirements need to be verified using hardware in the loop testing.  Test engineers would benefit from detailed guidance regarding HIL strategies, constraints, and parameters.

We encourage our clients to make more use of the ASPICE concept of verification criteria.  This holiday season, try giving the gift of verification criteria to your teams!