7 Examination of Vendor Practices for Configuration Management and Quality Assurance.

7.1 Introduction.

7.2 Scope and Basis of Examinations.

7.3 General Examinations Sequence.

7.3.1 Examination of Vendor Practices in Parallel with Other Qualification Testing.

7.3.2 Performance of Functional Configuration Audit as an Element of Integrated System Testing.

7.4 Examination of Configuration Management Practices.

7.4.1 Configuration Management Policy.

7.4.2 Configuration Identification.

7.4.3 Baseline, Promotion, and Demotion Procedures.

7.4.4 Configuration Control Procedures.

7.4.5 Release Process.

7.4.6 Configuration Audits.

7.4.7 Configuration Management Resources.

7.5 Examination of Quality Assurance Practices.

7.5.1 Quality Assurance Policy.

7.5.2 Parts & Materials Special Tests and Examinations.

7.5.3 Quality Conformance Inspections.

7.5.4 Documentation.

7 Examination of Vendor Practices for Configuration Management
and Quality Assurance........................................................................

7.1 Scope.......................................................................................................

7.2 Basis of Examinations...........................................................................

7.3 General Examinations Sequence.........................................................

7.3.1 Examination of Vendor Practices in Parallel
with Other Qualification Testing......................................................

7.3.2 Performance of Functional Configuration Audit
as an Element of Integrated System Testing................................

7.4 Examination of Configuration Management Practices.......................

7.4.1 Configuration Management Policy....................................................

7.4.2 Configuration Identification.................................................................

7.4.3 Baseline, Promotion, and Demotion Procedures............................

7.4.4 Configuration Control Procedures.....................................................

7.4.5 Release Process.................................................................................

7.4.6 Configuration Audits............................................................................

7.4.7 Configuration Management Resources............................................

7.5 Examination of Quality Assurance Practices.......................................

7.5.1 Quality Assurance Policy....................................................................

7.5.2 Parts & Materials Special Tests and Examinations........................

7.5.3 Quality Conformance Inspections......................................................

7.5.4 Documentation.....................................................................................

 

7                                                                                       Examination of Vendor Practices for Configuration Management and Quality Assurance

This section contains a description of the examination performed by the ITAs to confirm conformance with the requirements for configuration management and quality assurance of voting systems. It describes the scope and basis for the examinations, the general sequence of the examinations within the overall test process, and provides guidance on the substantive focus of the examinations.

ITAs shall design and perform procedures that examine documented vendor practices for quality assurance and configuration management as addressed by Volume I, Sections 7 and 8, and complemented by Volume II, Section 2.

Examination procedures shall be designed and performed by the ITA that address:

a.       Conformance with the requirements to provide information on vendor practices required by the Standards;

b.       Conformance of system documentation and other information provided by the vendor with the documented practices for quality assurance and configuration management.

The Standards do not require on-site examination of the vendor’s quality assurance and configuration management practices during the system development process. However, the ITAs conduct several activities while at the vendor site to witness the system build that enable assessment of the vendor’s quality assurance and configuration management practices and conformance with them. These include surveys, interviews with individuals at all levels of the development team, and examination of selected internal work products such as system change requests and problem tracking logs.

It is recognized that examinations of vendor practices, and determinations of conformance, entail a significant degree of professional judgement. These standards for vendor practices identify specific areas of focus for the ITAs, while at the same time relying on their expertise and professional judgement, as evaluated in the certification of the ITAs.

The specific procedures used by the ITA shall be identified in the Qualification Test Plan. Recognizing variations in vendors’ quality assurance and configuration management practices and procedures, the ITAs shall design examination procedures that account for these variations.

There is no required sequence for performing the examinations of quality assurance and configuration management practices. No other testing within the overall qualification testing process is dependent on the performance and results of these examinations. However, examinations pertaining to configuration management, in particular those pertaining to configuration identification, will generally be useful in understanding the conventions used to define and document the components of the system and will assist other elements of the qualification test process. 

While not required, ITAs are encouraged to initiate the examinations of quality assurance and configuration management practices early in the overall qualification testing sequence, and conduct them in parallel with other testing of the voting system. Conducting these examinations in parallel is recommended to minimize the overall duration of the qualification process,

As described in Volume I, Section 8, the functional configuration audit verifies that the voting system performs all the functions described in the system documentation. To help ensure an efficient test process, this audit shall  be conducted by ITAs as an element of integrated system testing that confirms the proper functioning of the system as a whole. Integrated system testing is described in more detail in Volume II, Section 6.

The examination of configuration management practices shall address the full scope of requirements described in Volume I, Section 8, and the documentation requirements described in Volume II, Section 2. In addition to confirming that all required information has been submitted, the ITAs shall determine the vendor’s conformance with the documented configuration management practices.

The ITAs shall examine the vendor’s documented configuration management policy to confirm that it:

a.       Addresses the full scope of the system, including components provided by external suppliers; and

b.       Addresses the full breadth of system documentation;

The ITAs shall examine the vendor’s documented configuration identification practices policy to confirm that they:

a.       Describe clearly the basis for classifying configuration items into categories and subcategories, for numbering of configuration items; and for naming of configuration items; and

b.       Describe clearly the conventions used to identify the version of the system as a whole and the versions of any lower level elements (e.g., subsystems, individual elements) if such lower level version designations are used.

The ITA shall examine the vendor’s documented baseline, promotion and demotion procedures to confirm that they:

a.       Provide a clear, controlled process that promotes components to baseline status when specific criteria defined by the vendor are met; and

b.       Provide a clear controlled process for demoting a component from baseline status when specific criteria defined by the vendor are met;

The ITA shall examine the vendor’s configuration control  procedures to confirm that they:

a.       Are capable of providing effective control of internally developed system components; and

b.       Are capable of providing effective control of components developed or supplied by third parties.

The ITA shall examine the vendor’s release process to confirm that it:

a.       Provides clear accountability for moving forward with the release of the initial system version and subsequent releases;

b.       Provides the means for clear identification of the system version being replaced;

c.       Confirms that all required internal vendor tests and audits prior to release have been completed successfully;

d.       Confirms that each system version released to customers has been qualified by a the appropriate ITA prior to release;

e.       Confirms that each system release has been received by the customer; and

f.        Confirms that each system release has been installed successfully by the customer;

The ITA shall examine the vendor’s configuration audit procedures to confirm that they:

a.       Are sufficiently broad in scope to address the entire system, including system documentation;

b.       Are conducted with appropriate timing to enable effective control of system versions; and

c.       Are sufficiently rigorous to confirm that all system documentation prepared and maintained by the vendor indeed matches the actual system functionality, design, operation and maintenance requirements.

The ITA shall examine the configuration management resource information submitted by the vendor to determine whether sufficient information has been provided to enable another organization to clearly identify the resources used and acquire them for use. This examination is intended to ensure that in the event the vendor concludes business operations, sufficient information has been provided to enable an in-depth audit of the system should such an audit be required by election officials and/or a law enforcement organization.

The examination of quality assurance practices shall address the full scope of requirements described in Volume I, Section 7, and the documentation requirements described in Volume II, Section 2. The ITA shall confirm that all required information has been submitted, and assess whether the vendor’s quality assurance program provides for:

a.       Clearly measurable quality standards;

b.       An effective testing program throughout the system development life cycle;

c.       Application of the quality assurance program to external providers of system components and supplies;

d.       Comprehensive monitoring of system performance in the field and diagnosis of system failures;

e.       Effective record keeping of system failures to support analysis of failure patterns and potential causes; and

f.        Effective processes for notifying customers of system failures and corrective measures that need to be taken, and for confirming that such measures are taken.

In addition to the general examinations described above, the ITA shall focus on the specific elements of the vendor’s quality assurance program indicated below.

The ITA shall examine the vendor’s quality assurance policy to confirm that it:

a.       Addresses the full scope of the voting system;

b.       Clearly designates a senior level individual accountable for implementation and oversight of quality assurance activities;

c.       Clearly designates the individuals, by position within the vendor’s organization, who are to conduct each quality assurance activity; and

d.       Provides procedures that determine compliance with, and correct deviations from, the quality assurance program at a minimum annually.

The ITA shall examine the vendor’s parts and materials special tests and examinations to confirm that they:

a.       Identify appropriate criteria that are used to determine the specific system components for which special tests are required to confirm their suitability for use in a voting system;

b.       Are designed in a manner appropriate to determine suitability; and

c.       Have been conducted and documented for all applicable parts and materials.

The ITAs shall examine the vendor’s quality conformance plans, procedures and inspection results to confirm that:

a.       All components have been tested according to the test requirements defined by the vendor;

b.      All components have passed the requisite tests; and

c.      For each test, the test documentation identifies:

1)      Test location;

2)      Test date;

3)      Individual who conducted the test; and

4)      Test outcome.

The ITAs shall examine the vendor’s voting system documentation to confirm that it meets the content requirements of Volume I, Section 7.5, and Volume I Section 2, and is written in a manner suitable for use by purchasing jurisdictions.