Section 4.5 Lifecycle, Quality Gates and Test Types

Figures 4.11 and 4.12 below depict the high-level view of the ICT project lifecycles (during this Engagement Phase) as described in Section 4.4.

These high-level views can assist in:

  • IV&V Engagement planning – which indicates the start and end of engagement.
  • Quality Gates planning – which indicate quality milestones.
  • test types planning – which indicate V&V efforts to be executed.


Figure 4.11:  Engagement Phase: ICT Project Lifecycle (Ground-Up, Legacy Disposed, Legacy Enhanced)


Figure 4.12:  Engagement Phase: ICT Project Lifecycle (Production System)

Part I: Quality Gates

In Figures 4.11 and 4.12, Part I: Quality Gates refer to the milestone checkpoints throughout the project where the IV&V Provider’s TMO does the following:

  • provides the IV&V reports as input(s) to The Agency’s Quality Gate Keeper,
  • puts forward recommendations on whether the activities may proceed towards the next Quality Gate.

To ensure synchronisation of the activities among the stakeholder (IV&V Provider’s TMO, The Agency’s PMO and Development Team), the IPP shall define the Quality Gates to be adhered to.

The Agency must perform the following before the IV&V Engagement begins:

  • establish the ICT project’s IPP,
  • establish the PMO-TMO relationship to be adopted,
  • establish the ICT project lifecycle of the project,
  • define the Quality Gates in IPP and the respective reports, as required
  • establish the need for QS Product Certification for COTS product acquisition, as required,
  • define the test types to be executed.

Part II: PMO

In Figures 4.11 and 4.12, Part II: PMO refers to The Agency’s PMO and its management of the ICT project, where typically The Agency’s ICT implementation project follows the sequential approach. 

Part III: TMO

In Figures 4.11 and 4.12, Part III: TMO illustrates The Agency’s Software Test Team augmented with IV&V Provider testing effort.  

Testing types highlighted in red are:

  • Tests typically performed by the IV&V Provider. However, the IV&V Provider may be engaged to perform any or all of the types of tests shown.
  • Tests to be reviewed and also augmented by the IV&V Provider if already performed by the Development Team.

4.5.1 Quality Gates

Table 4.9 until Table 4.12 below shows a list of typical IV&V activities by the IV&V Provider at the respective Quality Gate milestones. The tables explain IV&V activities at each gate and the expected ‘Quality Focused’ outcomes as well as the templates to be used.


Table 4.9:  Quality Gate 1 – Requirements


Table 4.10:  Quality Gate 2 – High-Level Design


Table 4.11:  Quality Gate 3 – System Testing (System Integration Testing)


Table 4.12:  Quality Gate 4 – Acceptance Testing

Throughout the development lifecycle, leading towards each Quality Gate,  the IV&V Provider team should strive to excel by performing the following:

  • Proactively participate in Quality Gates checkpoint meetings and provide recommendations throughout the development lifecycle,
  • Provide feedback early, especially during requirements gathering,
  • Work with The Agency’s in-sourced, co-sourced, or outsourced Development Team and keep the lines of communication open,
  • Provide objective feedback to eliminate surprises, by providing constant communication with Development Team,
  • Perform comprehensive, independent analysis with entry and exit criteria verified to minimize risks to the ICT project,
  • Continue to be a key participant working with The Agency teams.

Throughout the ICT  project lifecycle, the IV&V Provider may propose additional value adding services, such as:

  • assessing the Development Team’s processes and infrastructure using multiple criteria including:
    • statements-of-work,
    • The Agency’s adoption of relevant international standards,
    • the Development Team’s plans and policies.
  • evaluating the Development Team’s use of commercial or custom development and testing tools.
  • analysing the Development Team’s test plans for:
    • completeness and adequacy of coverage,
    • proper acceptance criteria,
    • sufficient planning of tools, facilities, procedures, methods, and resources,
    • adequate planning for regression testing, and
    • correct and complete traceability.
  • recommending changes to the test plans and processes when deficiencies are identified.
  • assessing results of requirements testing (requirements specifications review) and observing Development Team’s testing as directed by The Agency.
  • observing testing execution to confirm tests are conducted with approved test plans and procedures.
  • performing additional independent test assessments as directed by The Agency by generating test plans, test designs, test suites, and test procedures in preparation for IV&V Provider’s augmented testing.
  • providing the results of IV&V Provider’s testing to The Agency’s management, and to the Development Team.
  • submitting reports of anomalies detected to be logged by Development Team into development systems and monitored by the IV&V Provider to ensure its resolution and closure.
  • monitoring Service Level Agreements (SLAs) on defect resolution by the Development Team.
  • documenting resolutions and working with The Agency to monitor fixing of defects and subsequent retests.

4.5.2  Test Types

During testing, the Development Team must test the application to ensure that the proposed solution meets the defined and approved functional, technical, and quality requirements specifications. 

The Agency should use the IV&V Provider’s engagement to augment the tests conducted by the Development Team to ensure a quality product is released.  Depending on the project needs, the IV&V Provider may be engaged to perform:

  • mandatory minimum tests,
  • additional tests.

To determine the types of testing to be performed by the IV&V Provider, The Agency can determine the following: 

  • How good does the product need to be?
    • What do we measure?
    • How do we measure it?
    • How do we make a decision?
  • Are there any legal or compliance issues to be tested?
    • What industries will the acquired ICT be used in?
  • What will testing cost?
    • Time and resource (human and non-human resources).
  • What matters to The Agency’s users/customers who will be accessing the ICT?
    • What does the system do?
    • What are the Risks?
    • What are the ICT Requirements?

Mandatory minimum tests (for Production System ICT  project lifecycles)

For the mandatory minimum test, the IV&V Provider will be required to perform the following:

  • Independent assessments by reviewing and verifying the state of documentation for the system operating in production environment.
  • Independent assessment by reviewing The Agency’s capability to execute regression testing to complement software patches deployment.
  • To augment with the following test types:
    • Performance testing (load & stress testing),
    • Security (vulnerability) testing.
  • Deploy automated testing tools and test automation skills to deliver state-of-the-art services where required.  

Additional test services by IV&V Provider

Depending on the project’s needs, and regardless of lifecycle, the IV&V Provider may offer to perform the following additional tests:

  • multiple iterations of early testing,
  • risk-based approach of functional testing,
  • more than just the OWASP Top 10 vulnerability of security testing,
  • complex model of performance testing, including load and stress testing.

A host of additional value adding tests can ensure a high-quality ICT product is released for use. To finalise the additional tests required from the IV&V Provider, The Agency should seek the assistance of MyTCoE.

Table 4.9 below lists a selection of Testing Types for consideration.


Table 4.13:  Testing Types