|
|
Initiation responsibility: Agency/company management
Implementation responsibility: Head of Specialist Department, Head of IT Section
The test procedures described below are based on DIN ISO/IEC 12119 "Software Products, Quality Requirements and Testing Conditions", the Procedural Model for the Planning and Implementation of IT (V Model) and the Information Technology Security Evaluation Handbook (ITSEM), which are recommended as secondary literature.
Before deciding on a suitable software product, the products which were shortlisted according to the preselection (see S 2.81 Preselection of a Suitable Standard Software Product) must be obtained with a test licence and sufficiently tested. If it was not possible to test the product before purchase due to time limitations, internal procurement recommendations (adherence to internal standards) or any other reasons, tests must be conducted before the product is put into operation. The results of these tests then form the basis for the installation regulations and other approval conditions.
Although the product requirements were checked in the preselection stage on the basis of the manufacturers' declarations, it cannot be assumed that these requirements are met to the desired extent. Systematic testing must be conducted in order to check the suitability and reliability of the product on the basis of the Requirements Catalogue so that the most suitable product can then be selected.
It is recommended to divide these tests into four areas:
The test procedure of standard software is illustrated below.
On the basis of the shortlist drawn up during the preselection stage, those products should be selected which are to be tested. A test plan is then compiled.
This plan comprises the following:
These points are described in detail below.
Determining the contents of the test on the basis of the Requirements Catalogue
The requirements which are to be tested are selected on the basis of the Requirements Catalogue. In particular, these should be the features which are of great importance or which have a high confidence factor.
Checking references
Initial references were obtained during the preselection stage (see S 2.81 Preselection of a Suitable Standard Software Product). These can also be obtained if the external test group gives rise to sufficient confidence.
If a certificate was issued for the product in accordance with the criteria for the evaluation of the security of IT systems (ITSEC) or the Common Criteria (CC), the certification report should be used to check to what extent the test results can be taken into consideration.
An internal test can either be dispensed with or conducted on a small scale. The test capacities this leaves free can be distributed among other tests.
Determining the total testing time
In order to limit the time required for testing, the total time should be determined in advance, e.g. in working days or by setting a deadline.
Time planning, including time required for each test
When testing several products, it is recommended to run comparative tests. This means that all products are tested by one test group or in regard to one requirement of the Requirement Catalogue. The testing time should thus be determined for each requirement of the Requirement Catalogue and is thus automatically distributed evenly among all products to be tested. The testing time results from the testing depth and complexity of the feature. The testing depth of the various features should be based on the confidence factor, i.e. the amount of confidence placed in the correct operation of this feature. The proneness to error and frequency of use of the feature must also be taken into consideration. More detailed information is to be found in ISO 12119.
Notes:
The total testing time should then be distributed to the individual test sections in accordance with the relative testing time of the feature.
Determining persons-in-charge of testing
It should be determined for each test which tasks are to be carried out and who is responsible for these. In particular, it should be ensured that the staff / works council, the Data Privacy Officer and the IT Security Officer are involved in some tests.
Test environment
Testing is always destructive as errors are being looked for. Tests should thus always be conducted in an isolated test environment.
If possible, the test environment should be a precise functional copy of the production environment. It is generally not viable to completely recreate the production environment.
So that the same conditions are present for the selected products, a reference test environment should be defined. This can be further adjusted or limited for individual tests.
The resources required for the various tests (equipment, IT infrastructure) should be specified. It should be described in detail when, and to what extent, these must be available.
It is important that all operating systems in all versions used in production are available in the test environment. The intention is to determine system-based weaknesses of components of the production environment in connection with the standard software production to be installed. In exceptional cases, if aspects can be generalised, individual components can be omitted.
The following aspects should be observed and help to set up a reliable and suitable test environment:
After all planned tests have been concluded, it should be decided whether the test environment is to be dismantled. It may be necessary for more tests to be carried out, i.e. it might be viable to retain the test environment. Before the test environment is dismantled, the test data should be deleted if no longer required (e.g. for installation at a later date). Printer products should be disposed of correctly, programs should be deinstalled. The test licences of the products which were not selected should be returned.
Contents of the test documentation
The test plan should state how detailed the test documentation should be. The aspects of comprehensibility, reproducibility and completeness should be taken into consideration.
The test documentation must contain test plans, targets, processes and results. It must also describe the correspondence between the tests and the specified requirements. All test activities and the test evaluation (including reasons for decisions) should be set down in writing. These include:
The test group should be able to have access to clear documentation and records of the test activities and results (e.g. recording tool, forms etc.).
In the event that an automatic tool is used for testing, the test documentation must contain sufficient information about this tool and its usage so that the decision can be understood.
Determining criteria for a decision
When evaluating the contents of a test, the following three-point scale can be used, for example:
In the event that errors have arisen which cannot be reproduced, the tester must decide to which category (grade) the error should be allocated.
If errors have occurred which can be eliminated during testing, these should be tested again after elimination.
Example:
The example of a compression program in S 2.81 Preselection of a Suitable Standard Software Product is continued here in order to describe how the testing time can be determined for each requirement of the Requirement Catalogue. The testing time is based on the depth and complexity of the test. The confidence factor represents the amount of confidence in the feature.
The frequency of use, proneness to error and complexity of a feature are assessed as follows:
An exception to this is if an unchangeable feature of the product is to be examined which is independent of the proneness to error and frequency of use. In this case, the value 0 is given. This results in the following table for the compression program:
In this example, the testing time is defined as follows
and
(The percentage for the testing time in the last column of the table results from the values calculated for the testing time divided by the total of these values).
An example for another method of calculating the testing time and assessing the test results is described in ISO 12119. Here, the requirements are weighted as follows: Assessment of each test = (complexity + proneness to error) * (frequency of use + importance).
In any case, the person responsible for testing must decide on an adequate assessment method for both the product and the institution.
After drawing up a test plan, a tester or test group is appointed for each test specified in the test plan. The test group should be given the test plan and informed of the times for the individual tests.
Additional controls:
| © Copyright
by Bundesamt für Sicherheit in der Informationstechnik |
July 1999 |