The Open Group Verification Suite Development Process
Certified Products» Certification Register
Certification Guides» Open Brand Guide
» UNIX 03 Guide
» UNIX V7 Guide
Documents» Trademark License Agreement (TMLA)
» TMLA Signature page
» Trademark Usage Guide (TMUG)
» Fee Schedule
» Test Requirements
» Test Suites
» Product Standards
» Registration Forms
Conformance Statements» Blank Templates
Links» Problem Reports and Interpretations
» Test Suite Support
» COE Platform Certification
» General Testing Information
You are here: The Open Brand >Testing Test procedures >The Open Group Verification Suite Development Process
The Open Group has set of procedures for ensuring timely, quality, correctness, fitness-for-purpose test suite development . This brief paper gives an overview of the test development process.
The process for test development is based on experience gained through over twelve years of X/Open test development. This involves a multi-step process where objectivity, and traceability of test code to a written specification are the key. It is the written specification that is the definitive reference point and in cases of doubt assertions and tests are secondary and defer to the specification. Quality , correctness and fitness-for-purpose are achieved through a methodology that places regular checkpoints and peer-review feedback within the process.
The first step is development of a test specification. Secondly, a set of formal test assertions are developed. Thirdly, the test suite based on the specification and the assertions is developed and taken through a controlled development cycle before general availability. Lastly after completion of the test suite, the test suite is moved to support and maintenance mode where customer support questions, and interpretations are processed, and regular test suite updates made.
The Open Group has developed a set of criteria that the test specification, assertions and the test suite need to be meet in order to be accepted. At regular points formal reviews are held and feedback is obtained from the intended customer and specification owners to verify correct assumptions. A technical expert group is used to advise on specification interpretations during development (questions from developers) during Beta (issues raised by beta testers) and during test suite maintenance. This expert group is usually the owners of the specification. Such issues are dealt with in a way that ensures an archived set of referencable interpretations for other test suite users and potential brandees.
Test Suite Specification
A test suite specification is developed which provides a high level description of the test suite. This describes the test methodology, the functional coverage, the test architecture and states any assumptions about the test environment . It also includes a schedule of review points for the project.
Creation of a set of test assertions is a necessary step to bridge the gap between natural language specification and test suite code. The goal of the test suite code is to produce an executable representation of the requirements of the specification under test.
The Open Group adopts an assertion based technique for test specifications. This follows the IEEE Standard for Test Methods (1003.3). An assertion is developed for each definitive statement in the specification under test. Each assertion is designed to determine whether the statement is true or false for the implementation under test.
Typically, a definitive statement within a specification contains or implies a shall, should or may. A shall statement is an unconditional requirement. A should or may statement is a conditional requirement, based on support of an option. Some judgement needs to be exercised by the assertion writer to ensure that the assertions generated are based upon the requirements of specification, and not in some cases based on non-normative advice to programmers or such like.
For example, a specification might state:
Required files for /bin
This is written as 34 separate test assertions , for example
Reference 3.1-3 (C) If the implementation provides a C compiler: The XYZ command will exist in /bin. etc.The above example is a simple case, common examples might also include if-then-otherwise conditional tests etc. The key is to produce logical, testable statements that can be reasoned about, these are called test assertions. By producing test assertions, we can then produce tests (which maybe produced manually or if using Assertion Definition Language automatically). Typical test suites can contain thousands of test assertions.
Assertions also need to consider the level of testing. POSIX.3 defines three levels:
Testing occurs at the exhaustive level where practical. However in many areas it may not be practical to do so due to the time and system resources required. Most test suites perform thorough testing which often serves as a fair compromise in the coverage/cost ratio. Identification testing is rarely used unless the specification requires it, by stating merely that a feature exists and defining no behaviours or values.
Once the test suite specification has been developed it needs validating to assure that its a correct representation of the requirements of the specification under test.
The Open Group has developed methods for managing review feedback which allow development to proceed in an objective and timely fashion.
This is achieved using review guidelines and a pro-forma responses. Comments should be specific , and always include a specified action. Just stating that there is a problem, without stating a solution is not useful or procedurally acceptable.
The same applies for the test assertions.
At this point an iterative process occurs as test assertions are validated or shown to be false. This may be due to an incorrect assumption by the author of the test assertion, or in some cases what is now considered an error in the underlying specification. The latter should only be decided by the authors of the specifications, and not the test suite author, and these issues should be fed back to the specification owners so that corrective action can be taken. The test suite will note the issue in its output , but the final correct strategy cannot be undertaken until the specification owners have issued corrections or possibly a revised version of the specification.
Once a test suite has completed the development cycle The Open Group operates a formal process with guaranteed response times for resolving issues concerning the validity of individual tests. This process is confidential, objective, repeatable and fair, making use of industry consensus as appropriate. Where a test is found to give false 'fails' after all company and product details are removed, the formal interpretation is published on the open group web server for reference by other test suite support customers.
The Open Group has developed formal procedures for test acceptance. This involves up to two review cycles:
It is recommended that participants in the test program supply equipment and sample implementations for use by the Open Group during the development cycle. This can greatly reduce the effort for debugging and ensure maximum portability of the resulting test suite.
Test Suite Maintenance
Once test development is complete a test suite goes into a maintenance cycle. A support service is established using the world wide web and electronic mail. A database of sanitised interpretations is maintained that can be searched by other test suite users. Regular maintenance releases are produced up to twice a year if necessary, addressing customer issues identified by using the test suite live.