The LSB-FHS test suite, tests the filesystem hierarchy aspects of the Linux Standard Base, as defined in the FHS 2.0 specification.
The process for developing the test suite involves a multi-step process where objectivity, and relationship to a written specification are the key.
The first step is development of a test specification which should undergo formal review. Secondly, a test suite based on that specification. There are criteria for both the specification and the test suite that should be met (see criteria). At regular points there should be feedback from the customer audience / underlying specification owners to verify the correct assumptions.
In the case of the LSB-FHS test suite, the test specification is based on the FHS 2.0 specification, see www.pathname.com, as modified by the requirements placed by the LSB specification (for example LSB mandates presence of the X Window system which is optional in the FHS).
The test specification follows the POSIX 1003.3 guidelines and consists of a set of assertions , that is a set of statements which evaluate as true or false based on the specification under test. This can be thought of the translation between the natural language specification and the testable specification.
For example, the FHS has a statement in section 3.1:
Required files for /bin
The following commands have been included because they are essential.
A few are present because of their traditional placement in /bin.
{A list of 34 commands}
This is written as 34 separate test assertions , for example
Reference 3.1-3 (A) The implementation provides an exec-able version of the cat utility in the /bin directory. Result: PASS/FAIL Reference 3.1-4 (A) The implementation provides an exec-able version of the chgrp utility in the /bin directory. Result: PASS/FAIL etc.The above example is a simple case, more complex examples are for conditional tests etc. The key is to produce logical, testable statements that can be reasoned about, these are called test assertions. By producing test assertions, we can then produce tests.
In order for development to proceed in an objective fashion , review guidelines and a pro-forma response are used for getting feedback from the community. A one month review was held to check the correctness of the assertions with respect to the two specifications. The review group was the union of the lsb-spec and lsb-test mailing lists.
Review comments are then collated and then a set of proposed resolutions produced . Reviewers review the comments in the light of the specifications being tested and consider where applicable using one of the following pro-forma resolutions:
The review comments together with the proposed resolutions are then circulated to the review group for confirmation.
This allows test development to proceed based on the specifications. At this point an iterative process occurs as test assertions are validated or shown to be false. This may be due to one an incorrect assumption by the author of the test assertion, or in some cases what is now considered an error in the underlying specification. The latter should only be decided by the authors of the specifications, and not the test suite author, and these issues should be fed back to the specification owners so that corrective action can be taken. The test suite will note the issue in its output , but the final correct strategy cannot be undertaken until the specification owners have issued corrections or possibly a version of the specification.
The timeline of development has been as follows:
The following review instructions were used (these are based on the X/Open Aardvark bug eater format used for some specification development).
Comments should take the following format, which allows us to collate comments by section. Subject: BUG in LSB-FHS-TS_SPEC_V1.0 @ section , assertion_reference # Problem: Explain why here. Be sure to add sufficient explanation for someone not familiar with the problem to be able to make a decision. Action: Be specific, that is give precise editorial instructions for change. For example @ /foo, X.13-1 Problem: /foo/dp may not exist if the dp extension is not supported. The LSB spec says clearly that the dp extension is optional. Action: Make this a conditional test, replace the existing text with "If the implementation provides the dp extension then the /foo/dp directory exists and is searchable"