S 2.83 Testing Standard Software

Initiation responsibility: Head of Specialist Department, Head of IT Section

Implementation responsibility: Test group

The testing of standard software can be divided up into the preparation, implementation and evaluation. The following tasks must be carried out in these sections:

Test Preparation

• Determining the test methods for the individual tests (test type, processes and tools)

• Creating test data and test cases

• Establishing the necessary test environment

Performing the test

• Receipt tests

• Functional tests

• Tests of additional functional features

• Security-specific tests

• Pilot application

Test evaluation

The various tasks are described below

Test Preparation

Determining the test methods for the individual tests (test type, processes and tools)

Methods for carrying out tests are, for example, statistical analyses, simulation, proof of correctness, symbolic program execution, review, inspection, failure analysis. It should be noted that some of these test methods can only be carried out if the source coding is available. The suitable test method must be selected and determined in the preparation stage.

It must be clarified which processes and tools will be used for testing programs and checking documents. Typical processes for testing programs are, for example, black box tests, white box tests or penetration tests. Documents can be checked using informal methods, reviews or checklists, for example.

A black box test is a functionality test without knowledge of the internal program sequences. Here, the program is run with all data types for all test cases with troubleshooting and plausibility checks.

A white box test is a functionality test with disclosure of the internal program sequences, e.g. by source code evaluation or tracing. White box tests generally go beyond IT baseline protection and can not normally be carried out for standard software as the source code is not disclosed by the manufacturer.

Functionality tests are intended to prove that the test is in accordance with the specification. Using penetration tests, it is intended to determine whether known or assumed weaknesses can be exploited in practical operation, for example by attempts to manipulate the security mechanisms or by bypassing security mechanisms by manipulation at the operating system level.

The way the results are to be secured and evaluated should be stipulated, particularly as regards repeating tests. It should be clarified which data should be kept during and after the test.

Creating test data and test cases

The preparation of tests also includes the creation of test data. Methods and procedures should be stipulated and described in advance.

A number of test cases in accordance with the testing time must be created for each test. Each of the following categories should be taken into consideration.

Standard cases are cases which are used to test whether the defined functions are implemented correctly. The incoming data are called normal values or limit values. Normal values are data within the valid input area, limit values are threshold data.

Error cases are cases where attempts are made to provoke possible program error messages. The input values which should cause a predetermined error message to occur in the program are called false values.

Exceptional cases are cases where the program has to react differently than to standard cases. It must therefore be checked whether the program recognises these as such and then processes them correctly.

Examples:

• If the input parameters can be between 1 and 365, tests are to be carried out with false values (e.g. 0 or 1000), the limit values 1 and 365 and with normal values between 1 and 365.

• An appointment planning program should take national holidays into consideration. A special case is when a certain day is a holiday in all states except one. The program must then react appropriately for this state and this day.

In the event that it is too time-consuming or difficult to create test data, anonymous actual values can be used for the test. For reasons of confidentiality, actual data must be made anonymous. It should be ensured that these anonymous data do not cover all limit values and exceptional cases, these having to be created separately.

Beyond the test data, all types of possible user errors should be taken into consideration. Particularly difficult are all user reactions which are not planned in the program sequence and which are thus not correctly rejected.

Establishing the necessary test environment

The test environment described in the test plan must be established and the products to be tested installed. The components used should be identified and their configuration described. In the event that deviations from the described configuration arise when installing the product, this should be documented.

Performing the test

The test must be carried out using the test plan. Each action, together with the test results, must be adequately documented and evaluated. In particular, if errors appear, these must be documented in such a way that they can be reproduced. Operating parameters suited to later production working must be determined and recorded to enable installation instructions to be drawn up later.

If additional functions are detected in the product which are not listed in the Requirements Catalogue but can nevertheless be of use, a short test for them must be carried out at the very least. If it becomes apparent that this function is of particular importance for later operation, they must be tested in full. For the additional test expenditure incurred, application must be made if necessary for an extension of the time limit to the person responsible. The test results must be included in the overall evaluation.

If, when processing individual test contents, it becomes apparent that one or several requirements of the Requirements Catalogue were not sufficiently specific, these must be put in more specific terms if necessary.

Example: In the Requirements Catalogue, encryption is demanded to safeguard the confidentiality of the data to be processed. During testing it has become apparent that off-line encryption is unsuitable for the intended purpose. An addition must therefore be made to the Requirements Catalogue with regard to on-line encryption. (Off-line encryption must be initiated by the user and each of the elements to be encrypted must be specified; on-line encryption is carried out in a transparent way on behalf of the user with pre-set parameters.)

Receipt tests

Before all other tests, the following basic aspects must first be tested, as any failure in these receipt tests will lead to direct actions or the stopping of the test:

• The absence of computer viruses in the product must be checked by a current virus search program.

• It must be established in an installation test whether the product can be installed simply, completely and comprehensibly for the later-intended purpose. Likewise, there must be a check on how the product is completely de-installed.

• The running capabilities of the product must be checked in the planned usage environment; this comprises in particular a check of screen editing, printer output, mouse support, networking capability, etc.

• The completeness of the product (programs and manuals) must be checked, e.g. by comparing with the inventory, the product specification or similar.

• Short tests of program functions should be performed which are not explicitly mentioned in the requirements, with regard to function, plausibility, freedom from error, etc.

Functional tests

The functional requirements which were placed on the product in the Requirements Catalogue must be examined in terms of the following aspects:

• Existence of the function by calling up in the program and evaluation of the items of program documentation.

• Freedom from error or correctness of the function

In order to guarantee the freedom from error or correctness of the function, depending on the test level various test procedures should be used during the check such as black box tests, white box tests or simulated production running.

The test data and test cases created in the initial phase are used in the functionality test. During the functionality test it is necessary to compare the test results with the specified requirements. In addition, a check should be made on how the program reacts in the case of faulty input parameters or faulty operation. The function must also be tested with the limit values of the intervals of input parameters and with exceptional cases. These must be detected accordingly and correctly handled.

• Suitability of the function

The suitability of the function is distinguished by the fact that the function

• actually fulfils the task to the required extent and in an efficient manner and

• can be integrated easily into normal work processes.

If the suitability of the function is not obvious, the solution is to test this in a simulated production operation, but still in the test environment.

• Consistency

The consistency of the separate functions must be checked, in each case between the Requirements Catalogue, the documentation and the program. Any contradictions must be documented. Discrepancies between the documentation and the program must be recorded in such a way that they can be incorporated into the additions to the documentation when the product is used later.

Tests of additional functional features

The additional features itemised in the Requirements Catalogue alongside the security-specific features and the functional features must also be checked:

• Performance

Running time behaviour should be determined for all planned configurations of the product. In order to test performance adequately, general tests in which production working is simulated, or a pilot application with selected users, are useful. It must be established whether the set performance requirements are being met.

• Reliability

Behaviour during accidentally or maliciously caused system crashes (crash test) must be analysed and it must be established what damage results from this. A record must be made of whether the product can be properly and correctly restarted following system crashes. A check must also be made as to whether there can be direct access to data bases independent of the regular program function. In many cases such access can lead to loss of data and should be prevented by the product. It should also be recorded whether the program supports possibilities of reversing "critical actions" (e.g. deleting, formatting).

• User-friendliness

Whether the product is user-friendly depends, to a particular degree, on the subjective feeling of the tester. However, the following aspects can provide clues when making the assessment:

• technology of menu surfaces (pull-down menus, scrolling, drag & drop, etc.),

• design of menu surfaces (e.g. uniformity, comprehensibility, menu-driven operation),

• keyboard layout,

• error messages,

• trouble-free access to interfaces (batch operation, communication, etc.),

• readability of the user documentation,

• help functions.

Analysis of user-friendliness must describe possible modes of operation of the product, including operation following handling- or operating errors, and their consequences and implications for maintaining secure operation.

• Maintainability

Personnel and financial expenditure on the maintenance and care of the product should be determined during testing. This can be estimated with the aid, for example, of reference factors such as other reference installations, tests in specialist magazines, or using the installation expenditure determined during testing. To do this, the number of manual interventions which were necessary during installation to arrive at the configuration sought must be documented. If experience with preceding versions of the tested product has already been accumulated, an analysis should be made of how expensive their maintenance was.

Enquiries should be made regarding the extent to which support is offered by the manufacturer or seller and under what conditions. If a hotline is offered by the manufacturer or seller, its ease of access and quality should also be considered.

• Documentation

The existing documentation must be checked with a view to whether it is complete, correct and consistent. In addition to this it should be understandable, clear, error-free and easy-to-follow.

It must further be monitored whether it is adequate for secure use and configuration. All security-related functions must be described.

Over and above this, the following additional points of the Requirements Catalogue must be tested:

• compatibility requirements

• interoperability

• conformity to standards

• adherence to internal rules and legal provisions

• software quality

Security-specific tests

If specific security requirements were placed on the product, in addition to the trials mentioned above, the following aspects must be examined:

• effectiveness and correctness of the security functions,

• strength of the security mechanisms and

• absolute necessity and unavoidability of the security mechanisms.

As the basis for a security check the Manual for the Evaluation of the Security of Information Technology Systems (ITSEM) could, for example, be consulted. This describes many of the procedures shown below. The additional comments are an aid to orientation and serve as an introduction to the topic.

At the outset it must first be demonstrated by functional tests that the product supplies the required security functions.

Following this, it must be checked whether all the required security mechanisms were mentioned in the Requirements Catalogue and, if necessary, this must be amended. In order to confirm or reject the minimum strength of the mechanisms, penetration tests must be carried out. Penetration tests must be carried out after all other tests, as indications of potential weaknesses can arise out of these tests.

The test object or the test environment can be damaged or impaired by penetration tests. To ensure that such damage does not have any repercussions, backups should be made before penetration tests are carried out.

Penetration tests can be supported by the use of security configuration- and logging tools. These tools examine a system configuration and search for common flaws such as, for example, generally legible files and missing passwords.

Using penetration tests, the product should be examined for design flaws by employing the same methods a potential 'invader' would use to exploit weak points, such as, for example,

• changing the pre-defined command sequence,

• executing an additional function,

• direct or indirect reading, writing or modification of internal data,

• execution of data whose execution is not planned,

• use of a function in an unexpected context or for an unexpected purpose,

• activation of the error recovery,

• use of the delay between the time of checking and the time of use,

• breaking the sequence by interrupts, or

• generating an unexpected input for a function.

The mechanism strengths are defined using the terms specialised knowledge, opportunities and operating resources. These are explained in more detail in ITSEM. For example, the following rules can be used for defining mechanism strength:

• If the mechanism can be mastered by a lay person alone within minutes, it cannot even be classified as low.

• If a successful 'invasion' can be carried out by anyone except a lay person, the mechanism must be classified as low.

• If an expert is required for a successful 'invasion' and the expert takes some days with the available equipment, the mechanism must be classified as medium.

• If the mechanism can only be mastered by an expert with special equipment and the expert takes months to do it and has to come to a secret arrangement with a system manager, it must be classified as high.

It must be ensured that the tests carried out cover all specific security functions. It is important to note that only errors or differences from the specifications can ever be determined by testing, never the absence of errors.

Typical aspects of investigation can be shown by a number of examples:

Password protection:

• Are there passwords which have been pre-set by the manufacturer? Typical examples of such passwords are the product name, the manufacturer's name, "SUPERVISOR", "ADMINISTRATOR", "USER", "GUEST".

• Which file changes if a password was changed? Can this file be replaced by an old version from a backup to activate old passwords? Are the passwords stored in encrypted form or are they readable in plain text? Is it possible to make changes in this file to activate new passwords?

• Is access actually blocked following several incorrect password entries?

• Are programs offered in magazines or mailboxes which can determine the passwords of the product being examined? Such programs are available for some standard applications.

• If files are protected by passwords, can the position at which the password is stored be determined by a comparison of a file before and after the change in the password? Is it possible to enter changes or old values at this point in order to activate known passwords? Are the passwords stored in encrypted form? How is the position allocated if password protection is deactivated?

• Can the password testing routine be interrupted? Are there key combinations with which password entry can be bypassed?

Access rights:

• In which files are access rights stored and how are they protected?

• Can access rights be altered by unauthorised persons?

• Can files be inserted using old access rights and which rights are needed for this?

• Can the rights of the administrator be restricted such that he does not obtain access to the usage- or protocol data?

Data backup:

• Can backups which have been created be reconstructed without difficulty?

• Can backups be protected by a password? If so, can the password trial attempts described above be used?

Encryption:

• Does the product offer the possibility of encrypting files or backups?

• Are several different encryption algorithms offered? In this connection, generally speaking, the following rule of thumb should be observed: "The quicker an encryption algorithm produced in software is, the more insecure it is."

• Where are the keys used for encryption and decryption stored?

In the case of local storage there must be an examination of whether these keys are password-protected or are protected by a second encryption with a further key. In the case of password protection the above points must be taken into account. In the case of over-encryption, consideration must be given to how the accompanying key is protected.

In addition, the following points can be considered: which file changes if a key is changed? By comparing this file before and after the change in the code, the point can be determined at which this key is stored. Is it possible to make changes at this point to activate new keys which are then employed by the user, without the latter noticing the illicit change?

• Are there keys which have been pre-set by the manufacturer which have to be changed before the first use of the program?

• What happens if an incorrect key is entered during decryption?

• Following the encryption of a file, is the unencrypted variant deleted? If so, is it reliably overwritten? Is a check made before deletion as to whether the encryption was successful?

Logging:

• Is access to protocol data denied to unauthorised persons?

• Are the activities to be logged fully recorded?

• Does the administrator have the option, by virtue of his privileged rights, of obtaining access to protocol data without authorisation and unobserved, or can he deactivate the logging without being noticed?

• How does the program react if the logging memory overruns?

In addition to this it must be ascertained whether, as a result of the new product, security features will be circumvented elsewhere. Example: the product to be tested offers an interface to the operating system environment, previously however, the IT system was configured in such a way that no such interfaces existed.

Pilot application

Following the conclusion of all other tests a pilot application, i.e. use under real conditions, might still be considered necessary.

If the test is carried out in the production environment using actual data, the correct and error-free operating method of the program must have been confirmedto begin with a sufficient number of tests, in order not to jeopardise the availability and integrity of the production environment. For example, the product may be installed at the premises of selected users who will then use it for a set period in actual production conditions.

Test evaluation

Using the decision criteria specified, the test results must be assessed and all results must be assembled and submitted along with the test documentation to the procurer, or the person responsible for the test.

With the aid of the test results a final judgement should be made regarding a product to be procured. If no product has passed the test, consideration must be given as to whether a new survey of the market should be undertaken, whether the requirements set were too high and must be changed, or whether procurement must be dispensed with at this time.

Example:

Using the example of a compression program, one possibility is now described of evaluating test results. Four products were tested and assessed in accordance with the three-point scale derived from S 2.82 Developing a Test Plan for Standard Software.

Product 3 had already failed at the pre-selection stage and was therefore not tested.

Product 4 failed in the test section "correct compression and decompression", because the performance of the feature was assessed with a 0, although it is a necessary feature.

In calculating the assessment scores for products 1 and 2, the marks were used as multipliers for the respective significance coefficient and the total finally arrived at.

Product 1: 10*2+10*2+10*2+10*2+2*0+4*2+4*2+2*2 = 120

Product 2: 10*2+10*2+10*2+10*2+2*2+4*2+4*1+2*1 = 118

Following the test evaluation product 1 is thus in first place but is closely followed by product 2. The decision in favour of a product now has to be taken by the procurer using the test results and the price-performance ratio resulting from them.

Additional controls:

• Is the hardware- and software configuration used in conformity with the Requirements Catalogue?

• Do manufacturers or sellers offer support or maintenance services in connection with the use of the product?

• Are all functions relevant to the user described fully and comprehensibly in the user documentation?

• Do the existing documentation items contain a table of contents, a key word index and page references?

• Are all required functions executable and correct?

• Is the product reliable and robust in its usage environment? Under limit loads or in the event of faulty operation, can data be corrupted or destroyed?

• Are inadmissible and non-defined entries not processed in the same way as admissible ones?

• Were test documentation items produced in accordance with the standards?