Resources and responsibilities related to the test environment are important because if not handled properly, the tests and their results might be unreliable, and thus test results become null and void.
As part of the minimum conformance with this IV&V Handbook, considerations for the test environment must be addressed.
The test environment shall include the resources and responsibilities related to the:
- management and control over the test infrastructure and automation, which will be in line with the requirements of the specific ICT
- configuration management of IV&V Provider’s engagement artefacts and test wares.
- configuration management of the test basis such as Requirement Specifications, Design Specifications etc.
- configuration management of the System Under Test (SUT) developed and delivered by the Development Team.
- management and control of test data, test results, test analysis and reports.
- management of procedures for provisioning, SUT transmittal, defect reporting, change control, defects fixing, re-testing etc.
The IV&V Provider may or may not provide the infrastructure. Thus, The Agency need to specify who is responsible for test infrastructure resources and responsibilities in the RFP and tender documents. The Agency must specify:
- how they will support a test environment
- and what will be bought, loaned, or shared with respect to the test infrastructure
The IV&V Provider must be assigned and given the authority by The Agency to take charge of the following aspects of the test environment:
- Test provisioning with setting up the IV&V test environment before testing begins.
- Cleaning up the environment at the end of the IV&V tests.
- Protecting IV&V data and test data from sensitive sources.
- The overall security of the IV&V environment while running relevant tests.
- Any other matters related to ensuring the validity of the IV&V test results.
The test environment also includes test data, which is managed by the IV&V Provider during the Design and Build steps of the software development lifecycle.
Depending on the ICT project requirements, the IV&V Provider may seek The Agency’s approval to use data or database from the production system to be used as the IV&V test data. Examples of such situation are:
- functional testing involving complex business rules and process,
- performance testing on very large databases,
- testing involving integration with other production system(s),
- security testing on certain scenarios,
- etc.
Specifically for non-functional testing (such as Performance Test, Load Test, Stress Test etc.), The Agency is to finalise the need for the test environment which takes into account the “test infrastructure configuration” to “production infrastructure configuration” ratio.
In doing so, it will facilitate both the Development Team and IV&V Provider to objectively interpret the result of the performance test and subsequently identify appropriate fixes as required.