Version 2.0 6 November, 1997

(THIS DOCUMENT IS BASED ON THE PRE-EXISTING EQUIVALENT DOCUMENTS PUBLISHED BY NIST/ITL. CERTAIN LIMITED AND SPECIFIC CHANGES HAVE BEEN APPLIED TO THE TEXT TO REFLECT THE OPEN GROUP POLICIES. PLEASE CONTACT THE OPEN GROUP FOR FURTHER INFORMATION OR EXPLANATION OF THE DIFFERENCES BETWEEN THE NIST FIPS CERTIFICATION PROCESS, THE OPEN GROUP FIPS CERTIFICATION PROCESS AND THE OPEN BRAND)

TABLE OF CONTENTS

Program Summary

Summary of Process to Accomplish Validation

1. Introduction

1.1. Background

1.2. Purpose

1.3. Scope

1.4. Overview of Testing

2. Organisational Model for FIPS 151-2 Validation

2.1. FIPS 151-2 Certification Policy

2.2. Testing Laboratories

2.3. Clients

3. AFTL Accreditation

4. Testing Framework

4.1. Conformance Testing

4.2. Testing Program Implementation and Administration

4.3. Test Method

4.4. Validation Certificate

4.4.1. Issuance

4.4.2. Content

5. Procedures for Registration

6. Recognition of Other Accreditation Testing Activities

7. References

Appendix A - Terms and Abbreviatiions

Appendix B - Test Methods for FIPS 151-2

Appendix C - The Open Group FIPS 151-2 Certifacation Fees

Appendix D - FIPS 151-2 Conformance Test Suites

Program Summary

The Information Technology Laboratory (ITL) of the National Institute of Standards and Technology (NIST) developed standards, provided technical assistance, and carried out research to advance the effective use of computers by government and industry. NIST/ITL worked through voluntary industry standards organizations to develop standards that will met the needs of government users. These standards were issued as Federal Information Processing Standards (FIPS).

This document is intended to inform Government agencies, industry, standards development bodies, and other interested organizations of the policy and procedures related to conformance testing for The Open Group FIPS 151-2 certification. A summary of the conformance testing process is given below. This summary is a brief synopsis of the steps involved to achieve the conformance testing of a product.

A previous version of this document included the option of utilizing VSX4 as a test suite suitable for the Open Group FIPS certification program. After discussion with NIST/ITL this option has been withdrawn. Thus the only route to FIPS 151-2 certification is through testing using the PCTS in a test laboratory accredited for this purpose. Non accredited self testing or the use of VSX4 is not permitted if certification is required. In this respect the Open Group FIPS 151-2 certification program is essentially the same as that which was run by NIST/ITL.

Customers are invited to make use of the FIPS mode of VSX4 for their development testing prior to application for formal certification. The results however of such testing will have no effect on the success or otherwise of that certification.

SUMMARY OF PROCESS TO ACCOMPLISH VALIDATION

1 Client contacts The Open Group for list of Accredited FIPS 151-2 Testing Laboratories (AFTLs)

2 Client contacts an AFTL. Defines services needed.

3 Client and AFTL agree on services. Contract specifying fees and services.

4 Client provides product to test. Product X and documentation.

5 AFTL provides required services. AFTL provides NIST-PCTS and expertise.

6 If an The Open Group FIPS 151-2 Certificate is the object of testing, Test results and documentation are sent to The Open Group for evaluation.

7 The Open Group evaluates the test results and documentation.

8 If test report and documentation is acceptable, The Open Group issues a FIPS 151-2 Certificate of Validation for Product X on implementation tested.

1. INTRODUCTION

1.1 Background

The National Institute of Standards and Technology (NIST), Information Technology Laboratory (ITL) was responsible for developing U.S. Government-wide Standards for computer software, hardware, data management, networks and security, and related telecommunication systems. The authority for this responsibility was assigned under Federal Property and Administrative Services Act of 1949,as amended; Public Law 89-306 (79 Stat. 1127); Executive Order 11717 (38 FR 12315, dated May 11, 1973); Part 6 of Title 15 Code of Federal Regulations (CFR); and Public Law 100-235.

NIST/ITL developed standards, provided technical assistance, and carried out research to advance the effective use of computers by government and industry. NIST/ITL worked through voluntary industry standards organizations to develop standards that will meet the needs of government users. These standards are issued as Federal Information Processing Standards (FIPS) and provided the foundation for compatibility and, where necessary, interoperability among government systems implementing these standards. FIPS also served as the basis for Government acquisition of commercial off-the-shelf products and services from competitive sources. NIST/ITL was responsible for organizing, managing, directing and administering the FIPS program.

The pace of standards development for information systems (information processing and telecommunications) has intensified in recent years, stimulated by user needs for inter connectivity of hardware, software, and network systems. These standards are increasingly complex, often describing functional requirements and allowing for numerous options in implementation.

To achieve interoperability and effective use of information systems, users need off-the-shelf products that work together and conform to these emerging standards. Where products are expected to support complex standards specifications, conformance testing may be required to reduce risks and raise consumer confidence in information system products.

In 1997 the NIST/ITL announced its intention to cease the operation of its FIPS 151-2 certification program by the end of that year. The Open Group announced its intention to offer a FIPS 151-2 Certification service from October 1, 1997 based on the NIST/ITL program.

1.2 Purpose

This document is intended to inform Government agencies, industry, standards development bodies, and other interested organizations of The Open Group policy with regard to conformance testing of products to the FIPS 151-2 standard.

The objectives of The Open Group FIPS 151-2 Testing Policy are:

• To reduce the overall information systems costs by making it easier and less expensive to maintain information technology applications and to transfer these applications among different FIPS 151-2 conforming information systems, including replacement systems.
• To protect the technical assets and staff time of the Federal Government by insuring to the extent possible that products (off-the-shelf or government developed) brought into the Federal inventory comply with Government approved FIPS.
• To identify test methods and competent testing laboratories for assisting Government agencies in the procurement of industry supplied FIPS 151-2 conformant products.
• To ensure the portability and interoperability of FIPS 151-2 conformant products.

1.3 Scope

This document defines policy and procedures related to conformance testing for FIPS 151-2. Testing for The Open Group Brand is not addressed.

In determining testing requirements for FIPS 151-2, a number of areas are considered: Government testing needs, test method technology, standard specifications, alternative testing sources (third-party testing, Government testing, self-testing, etc.), and existing accreditation and certification systems.

The policy and procedures for conformance testing defined herein apply whenever the FIPS 151-2 standard is required to support Government objectives for information systems.

This document is addressed to:

• Agencies of the Federal Government intending to procure FIPS 151-2 products,
• Suppliers of FIPS 151-2 products wishing to market to the Federal Government, and
• Suppliers of FIPS 151-2 test services wishing to sell their services as an AFTL.

1.4 Overview of Testing

This document is concerned with conformance testing from the point of view of both the conduct of the test and the evaluation of the tester's capability; drawing a distinction between testing on the one hand, and accreditation on the other.

Accreditation is the administrative act of recognizing that a testing laboratory is qualified to conduct conformance testing, having met specific technical and organizational criteria. (see Section 3.)

Certification is the administrative act of recognizing that testing has demonstrated conformance to the standard, and of publicly registering the results.

2 ORGANIZATIONAL MODEL FOR FIPS 151-2 VALIDATION

Conformance testing for FIPS 151-2 will be accomplished in accordance with the organizational model described in The Open Group "FIPS 151-2 Conformance Testing Policy and Procedures" (see References). This model consists of a certification authority, testing laboratories, and clients.

2.1 FIPS 151-2 Certification Policy

The Open Group Conformance Quality Manager provides the overall direction for organizing, managing, directing, and administering the FIPS 151-2 The Open Group conformance testing and certification program and is the FIPS 151-2 Certification Authority for this program.

The FIPS 151-2 Certification Authority:

1. establishes and maintains the FIPS 151-2 conformance testing program policies and procedures,
2. approves the test methods used in determining compliance of products,
3. Insures that facilities are available to maintain the test methods,
4. evaluates and resolves disputes on all matters concerning conformance testing,
5. with the National Voluntary Laboratory Accreditation Program (NVLAP), establishes the accreditation criteria for testing laboratories for each test method as appropriate,
6. through NVLAP, ensures that there are procedures for recommending the laboratory either be granted or denied accreditation for use of a specific test method, as defined in this document and in The Open Group Branding documents. This recommendation is based on a review of the evaluation and other records to ensure that all technical, financial, and administrative obligations have been satisfied. (See NVLAP Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing),
7. maintains and publishes a list of testing laboratories recognized by NVLAP (or equivalent organization) to perform conformance testing for a specific test method.
8. establishes the fees or rates for The Open Group provided products and services,
9. co-ordinates with other certification authorities to review certification criteria for the purpose of harmonizing certificates and making provisions for mutual recognition of certification,
10. develops and maintains the procedures to be followed by clients of testing laboratories in order to receive a Certificate of Validation,
11. if test results are satisfactory, issues a Certificate of Validation based on the evaluation of test reports and accompanying documentation,
12. establishes the effective date of the Certificate of Validation,
13. maintains and periodically publishes a register of products that have a Certificate of Validation,
14. periodically assesses the need for a conformance testing program, maintenance of test methods or testing laboratory accreditation program.

2.2 Testing Laboratories

"AFTL's", as used in this document, refers to accredited testing facilities as described in the related NVLAP document Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing. AFTL's are accredited to test conformity to FIPS 151-2 utilizing the NIST PCTS. Such Testing Laboratories (AFTLs) will:

1. obtain and maintain laboratory accreditation as appropriate,
2. conduct conformance testing in accordance with the prescribed procedures in Section 4.3,
3. prepare test results in accordance with the appropriate The Open Group FIPS 151-2 Testing Policy Certificate of Validation Requirements,
4. conform to NVLAP guidelines in the NVLAP Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing,
5. pay all relevant fees,
6. participate in training sessions or meetings if required by The Open Group and be up-to-date on changes to the test method and the conformance testing procedures,
7. provide feedback to The Open Group on problems and improvements relating to the test methods and conformance testing procedures,
8. treat all test results and documents confidentially, except those which are explicitly stated as public,
9. comply with existing laws (accreditation does not relieve the AFTL of the need to observe and comply with existing Federal, State, and local statutes, ordinances, or regulations that may be applicable to its operation, including consumer protection and antitrust laws),
10. have the right to publish, within specified limits, their accredited status. The major restriction is that advertising must not imply product certification by The Open Group, NIST or the U.S. Government. (See advertising restrictions in the NVLAP Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing.)

2.3 Clients

The responsibilities of a client include:

1. unless otherwise agreed to by the AFTL, provide the test facilities and materials necessary for testing,
2. providing a FIPS 151-2 conforming product to be tested, and
3. providing documentation for the product being tested.

3. AFTL ACCREDITATION

The Open Group will carry out its responsibilities for conformance certification through testing laboratories judged to be competent to objectively utilize the NIST PCTS. The Open Group will draw upon NVLAP as the basis for accrediting laboratories. If the AFTL is to be based outside of the United States of America the applicant laboratory may contact the Open Group Conformance Quality Manager to discuss whether suitable alternative accreditation bodies are locally available or the alternative of direct accreditation by the Open Group

The U.S. Department of Commerce, administers NVLAP. "NVLAP's function is to accredit public and private testing laboratories based on evaluation of their technical qualifications and competence for conducting specific test methods in specified fields of testing." (See NVLAP Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing.) For further information about NVLAP, or for assistance in understanding and meeting the NVLAP requirements and criteria, contact NVLAP directly.

National Voluntary Laboratory Accreditation Program

National Institute of Standards and Technology

Bldg. 411 Room A124

Gaithersburg, MD 20899

Phone: (301) 975-4016 FAX: (301) 975-3839

4. TESTING FRAMEWORK

4.1 Conformance Testing

All testing will be done using the appropriate FIPS 151-2 Conformance Test Suite (NIST-PCTS) developed by NIST/ITL. These test suites are available from the National Institute of Standards and Technology (NIST), a branch of the Department of Commerce in Gaithersburg Md(see Appendix C for details).

4.2 Testing Program Implementation and Administration

Listed below are highlights of the testing program implementation and administration.

• The NIST-PCTS is available for a fee from NIST for public and Government distribution. (See Appendix C).
• If necessary The Open Group will provide "fixes" via its web site to the original NIST PCTS as required to support its FIPS 151-2 certification Programme. The scope of these fixes will be limited to the correction of errors in the test suite.
• All FIPS 151-2 conformance testing using the NIST PCTS will be accomplished by AFTLs.
• There will be a certification fee charged to cover administrative costs of evaluating the test report. (See Appendix D).
• The certificate of validation will be issued by The Open Group through the AFTL.
• Disputes or problems should be brought to the attention of The Open Group for resolution.

Interpretations of FIPS 151-2, are based on the procedures described in The Open Group Web site http://www.opengroup.org/testing/branding/ for interpretations for the Open Brand. There are however certain differences in the case of FIPS 151-2 certification which are defined in detail in "The Open Group FIPS 151-2 Testing Policy-Certificate of Validation Requirements for FIPS 151-2"

4.3 Test Method

There will be specific test requirements for each Federal standard related to FIPS 151-2. A list of acceptable test suites for FIPS 151-2 is provided in Appendix E. Sources of test method descriptions are listed in Appendix B.

4.4 Validation Certificate

4.4.1 Issuance

A Certificate of Validation will be issued by the Certification Authority when the following criteria have been met:

• the test has been conducted by an AFTL.
• the test results have been evaluated by the Certification Authority and meet the criteria listed for the test method (see Appendix B),
• the certificate has been requested by the Client, and
• appropriate fees have been paid to the Certification Authority for its evaluation efforts (see Appendix D).

4.4.2 Content

Articles in the Certificate of Validation shall include at least the following: date issued, product tested, test environment, reference standard, testing laboratory, and test method.

5. PROCEDURES FOR REGISTRATION

Essential to the operation of accreditation and certification is the maintenance of registers of test suites, accredited laboratories, and successfully tested products. These registers are maintained by The Open Group and are categorized as follows.

FIPS 151-2 Conformance Test Suites

Each test suite recognized by The Open Group for FIPS 151-2 conformance testing will be made available to the public and designated as the reference test suite (see Appendix F). The referenced test suite may be updated from time to time by The Open Group to:

• Correct any bugs that have been discovered, and to
• Include additional requirements as specified by a revised FIPS.

These available test suites will be published in this document.

Accredited Laboratories

Any testing laboratory which complies with the provisions of this document and the "NVLAP Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing", is added to the register of accredited laboratories. (see References)

Validated FIPS 151-2 Products

Products which have been issued The Open Group FIPS 151-2 Certificates of Validation are added to the register of tested products. (see References)

6. RECOGNITION OF OTHER ACCREDITATION TESTING ACTIVITIES

The Open Group will seek to provide adequate conformance testing for FIPS 151-2. In meeting this objective, The Open Group will consider the use of existing test methods, conformance testing procedures, testing laboratories and certification systems. It is not the intent of The Open Group to duplicate conformance testing activities where those activities meet The Open Group requirements. Thus The Open Group will co-ordinate with other organizations to harmonize conformance testing requirements.

The Open Group will base its recognition of test laboratories to the criteria defined in the "X/Open Laboratory Recognition Program". The procedures of that program will be used for direct accreditation of AFTL's if necessary.

7. REFERENCES

1. "Conformance Testing Policy and Procedures," (Proposed by the National Computer Systems Laboratory), Federal Register, Aug. 3, 1988.
2. "IEEE Standard Portable Operating System Interface for Computer Environments," IEEE Std 1003.1-1988, IEEE Computer Society, Institute of Electrical and Electronic Engineers, New York, NY, 1988.
3. "IEEE 1003.3 Standard for Test Methods for Measuring Conformance to FIPS 151-2," Draft 10.0, IEEE Computer Society, Institute of Electrical and Electronic Engineers, New York, NY, May 3, 1989.
4. "Interpretation Procedures for Federal Information Processing Standards for Software," Federal Information Processing Standard Publication 29-2 (FIPS 29-2), National Bureau of Standards, Gaithersburg, Md., September 14, 1987.
5. "Interpretation Procedures for Federal Information Processing Standards for Software," Federal Information Processing Standard Publication 29-2 (FIPS 29-2), National Bureau of Standards, Gaithersburg, Md., September 14, 1987.
6. "FIPS 151-2 Testing Laboratories and Validated Products," Computer Systems Laboratory, National Institute of Standards and Technology, Gaithersburg, MD (updated as products are validated).
7. Laboratory accreditation authority: National Voluntary Laboratory Accreditation Program National Institute of Standards and Technology Bldg. 411 Room A124 Gaithersburg, MD 20899
8. FIPS 151-2 Certification Authority: FIPS 151-2 Certification Authority National Institute of Standards and Technology Building Room B266 Gaithersburg, MD 20899
9. "NVLAP Directory of Accredited Laboratories," National Voluntary Laboratory Accreditation Program, National Institute of Standards and Technology, Gaithersburg, MD.
10. "FIPS 151-2: Portable Operating System Interface for Computer Environments," Federal Information Processing Standard Publication 151-1 (FIPS 151-1), National Institute of Standards and Technology, March 28, 1990.
11. "NVLAP Program Handbook, Computer Applications Testing, FIPS 151-2 Conformance Testing," National Voluntary Laboratory Accreditation Program, National Institute of Standards and Technology, Gaithersburg, MD, NISTIR 4522, March 1991.
12. "Validated Products List," Judy B. Kailey, editor, Software Standards Validations Group, National Institute of Standards and Technology, Gaithersburg, MD, issued quarterly.

Appendix A - Terms and Abbreviations

Accreditation. Administrative act of recognizing that a testing laboratory is qualified to conduct conformance testing, having met specific technical and organizational criteria.

Approved Test Methods. An organized system under which, on a uniform and equitable basis, products or services may be certified to meet specified standards.

AFTL. Accredited FIPS 151-2 Testing Laboratory.

Assessors. Experts selected by NVLAP to conduct an on-site assessment of a particular laboratory for the purpose of accreditation.

Certificate of Validation. A document attesting that a product or a service is in conformance with specific standards or technical specifications as determined through use of a specified test method.

Certification Authority. The Open Group Conformance Quality Manager provides the overall direction for organizing, managing, directing, and administering the FIPS 151-2 testing program.

Client. As used in this plan, Client refers to any organization or person who requires FIPS 151-2 conformance testing for any purpose.

Conformance. The state of an implementation satisfying the requirements and specifications of a specific standard as tested by a test suite.

FIPS. Federal Information Processing Standard as specified in a FIPS publication.

FIPS 151-1. Federal Information Processing Standard Publication 151-1, "FIPS 151-2: Portable Operating System Interface for Computer Environments".

ITL. Information Technology Laboratory (within NIST).

NIST. National Institute of Standards and Technology (formerly National Bureau of Standards (NBS)).

FIPS 151-2. The colloquial name for FIPS 151-2 related Federal Information Processing Standards (FIPS).

NVLAP. National Voluntary Laboratory Accreditation Program (within NIST).

PCTS. NIST FIPS 151-2 Conformance Test Suite.

POSIX. The colloquial name for the collection of IEEE 1003 Standards, the first of which is IEEE Standard Portable Operating System Interface for Computer Environments, IEEE Std. 1003.1-1988.

Test Suite. A complete set of tests necessary to perform conformance testing for a target system, together with the information and instructions needed to run the tests.

Appendix B - Test Methods for FIPS 151-2.

The descriptions of the test methods for FIPS 151-2 are issued under separate cover.

Appendix C - The Open Group FIPS 151-2 Certification Fees.

The Open Group administrative fee for the evaluation of test results in connection with FIPS 151-2 certification is $1000. For existing VSX test suite licensees the fee will be waived during the first year of the operation of the progam. Any fees due shall be submitted by AFTL with the test results to be evaluated.

Cheques should be made payable to The Open Group.

Appendix D - FIPS 151-2 Conformance Test Suites.

Test Suites acceptable for Conformance Testing for FIPS 151-2:

1. NIST-PCTS:151-2 (for testing conformance to FIPS 151-2 and reference standard ISO/IEC 9945-1).

VSX4 may be used for development purposes but is not acceptable as an indicator for compliance for FIPS 151-2 certification