4 Hardware Testing.......................................................................................

4.1 Scope.........................................................................................................

4.2 Basis of Hardware Testing........................................................................

4.2.1 Testing Focus and Applicability..............................................................

4.2.2 Hardware Provided by Vendor................................................................

4.3 Test Conditions..........................................................................................

4.4 Test Log Data Requirements....................................................................

4.5 Test Fixtures..............................................................................................

4.6 Non-operating Environmental Tests..........................................................

4.6.1 General...................................................................................................

4.6.1.1 Pretest Data.........................................................................................

4.6.1.2 Preparation for Test.............................................................................

4.6.1.3 Mechanical Inspection and Repair.......................................................

4.6.1.4 Electrical Inspection and Adjustment..................................................

4.6.1.5 Operational Status Check...................................................................

4.6.1.6 Failure Criteria.....................................................................................

4.6.2 Bench Handling Test..............................................................................

4.6.2.1 Applicability..........................................................................................

4.6.2.2 Procedure............................................................................................

4.6.3 Vibration Test..........................................................................................

4.6.3.1 Applicability..........................................................................................

4.6.3.2 Procedure............................................................................................

4.6.4 Low Temperature Test...........................................................................

4.6.4.1 Applicability..........................................................................................

4.6.4.2 Procedure............................................................................................

4.6.5 High Temperature Test...........................................................................

4.6.5.1 Applicability..........................................................................................

4.6.5.2 Procedure............................................................................................

4.6.6 Humidity Test..........................................................................................

4.6.6.1 Applicability..........................................................................................

4.6.6.2 Procedure..........................................................................................

4.7 Environmental Tests, Operating..............................................................

4.7.1 Applicability...........................................................................................

4.7.2 Temperature and Power Variation Tests.............................................

4.7.3 Power Disturbance...............................................................................

4.7.4 Electromagnetic Radiation...................................................................

4.7.5 Electrostatic Disruption........................................................................

4.7.6 Electromagnetic Susceptibility.............................................................

4.7.7 Electrical Fast Transient.......................................................................

4.7.8 Lightening Surge...................................................................................

4.7.9 Conducted RF Immunity.......................................................................

4.7.10 Magnetic Fields Immunity...................................................................

4.7.11 Accept/Reject Criteria.........................................................................

4.7.12 Data Accuracy....................................................................................

4.7.13 Maintainability Test..............................................................................

4.7.14 Reliability Test.....................................................................................

4.7.15 Availability Test...................................................................................

 

4                                                                                       Hardware Testing

This section contains a description of the testing to be performed by the ITAs to confirm the proper functioning of the hardware components of a voting system submitted for qualification testing. It describes the scope and basis for functionality testing, required test conditions for conducting hardware testing, guidance for the use of test fixtures, test log data requirements, and test practices for specific non-operating and operating environmental tests.

This section addresses the focus and applicability of hardware testing, and specifies the vendor’s obligations to produce hardware to conduct such tests.

ITAs shall design and perform procedures that test the voting system hardware requirements identified in Volume I, Section 3. Test procedures shall be designed and performed by the ITA for both operating and non-operating environmental tests:

¨       Operating environmental tests apply to the entire system, including hardware components that are used as part of the voting system telecommunications capability.

¨       Non-operating tests apply to those elements of the system that are intended for use at poll site voting locations, such as voting machines and precinct counters. These tests address environmental conditions that may be encountered by the voting system hardware at the voting location itself, or while in storage or transit to or from the poll site.

Additionally, compatibility of this equipment with the voting system environment shall be determined through functional tests integrating the standard product with the remainder of the system.

All hardware components custom-designed for election use shall be tested in accordance with the applicable procedures contained in this section. Unmodified COTS hardware will not be subject to all tests. Generally such equipment has been designed to rigorous industrial standards and has been in wide use, permitting an evaluation of its performance history. To enable reduced testing of such equipment, vendors shall provide the manufacturers specifications and evidence that the equipment has been tested to the equivalent of the Standards.

The specific testing procedures to be used shall be identified in the Qualification Test Plan prepared by the ITA. These procedures may replicate testing performed by the vendor and documented in the vendor’s TDP, but shall not rely on vendor testing as a substitute for hardware testing performed by the ITA.

The hardware submitted for qualification testing shall be equivalent, in form and function, to the actual production versions of the hardware units. Engineering or developmental prototypes are not acceptable unless the vendor can show that the equipment to be tested is equivalent to standard production units in both performance and construction.

Qualification tests may be performed in any facility capable of supporting the test environment. Preparation for testing, arrangement of equipment, verification of equipment status, and the execution of procedures shall be witnessed by at least one independent, qualified observer who shall certify that all test and data acquisition requirements have been satisfied.

When a test is to be performed at “standard” or “ambient” conditions, this requirement shall refer to a nominal laboratory environment at prevailing atmospheric pressure and relative humidity.

Otherwise, all tests shall be performed at the required temperature and electrical supply voltage, regulated within the following tolerances:

a.       Temperature of +/- 4 degrees F; and

b.       Electrical supply voltage +/- 2 VAC.

The ITA shall maintain a test log of the procedure employed. This log shall identify the system and equipment by model and serial number. Test environment conditions shall be noted.

In the event that the ITA deems it necessary to deviate from requirements pertaining to the test environment, the equipment arrangement and method of operation, the specified test procedure, or the provision of test instrumentation and facilities, the deviation shall be recorded in the test log. A discussion of the reasons for the deviation and the effect of the deviation on the validity of the test procedure shall also be provided.

The use of test fixtures or ancillary devices to facilitate hardware qualification testing is encouraged.  These fixtures and devices may include arrangements for automating the operation of voting devices and the acquisition of test data.

The use of a fixture to ensure correctness in casting ballots by hand is recommended. Such a fixture may consist of a template, with apertures in the desired location, so that selections may be made rapidly. Such a template will eliminate or greatly minimize errors in activating test ballot patterns, while reducing the amount of time required to cast a test ballot.

For systems that use a light source as a means of detecting voter selections, the generation of a suitable optical signal by an external device is acceptable. For systems that rely on the physical activation of a switch, a mechanical fixture with suitable motion generators is acceptable.

To speed up the process of testing and to eliminate human error in casting test ballots the tests may use a simulation device with appropriate software. Such simulation is recommended if it covers all voting data detection and control paths that are used in casting an actual ballot. In the event that only partial simulation is achieved, then an independent method and test procedure must be used to validate the proper operation of those portions of the system not tested by the simulator.

If the vendor provides a means of simulating the casting of ballots, the simulation device is subject to the same performance, reliability, and quality requirements that apply to the voting device itself so as not to contribute errors to the test procerss.

This section addresses a range of tests for voting machines and precinct counters, as such devices are stored between elections and are transported between the storage facility and polling site.

Environmental tests of non-operating equipment are intended to simulate exposure to physical shock and vibration associated with handling and transportation of voting equipment and precinct counters between a jurisdiction’s storage facility and precinct polling site. These tests additionally simulate the temperature and humidity conditions that may be encountered during storage in an uncontrolled warehouse environment or precinct environment.  The procedures and conditions of these tests correspond generally to those of MIL-STD-810D, “Environmental Test Methods and Engineering Guidelines,” 19 July 1983. In most cases, the severity of the test conditions has been reduced to reflect commercial, rather than military, practice.

Systems exclusively designed with system-level COTS hardware whose configuration has not been modified in any manner and are not subjected to this segment of hardware testing. Systems made up of individual COTS components such as hard drives, motherboards, and monitors that have been packaged to build a voting machine or other device will be required to undergo the hardware testing.

Prior to each test, the equipment shall be shown to be operational by means of the procedure contained in Subsection 4.6.1.5.  The equipment may then be prepared as if for actual transportation or storage, and subjected to appropriate test procedures outlined. After each procedure has been completed, the equipment status will again be verified as in Subsection 4.6.1.5.

The following requirements for equipment preparation, functional tests, and inspections shall apply to each of the non-operating test procedures.

The test technician shall verify that the equipment is capable of normal operation. Equipment identification, environmental conditions, equipment configuration, test instrumentation, operator tasks, time-of-day or test time, and test results shall be recorded.

The equipment shall be prepared as for shipping or storage, with any protective enclosures or internal restraints normally used for transportation and handling between the storage facility and the polling location.

After the test has been completed, the devices shall be removed from their containers, and any internal restraints shall be removed.  The exterior and interior of the devices shall be inspected for evidence of mechanical damage, failure, or dislocation of internal components.  Devices shall be adjusted or repaired, if necessary.

After completion of the mechanical inspection and repair, routine electrical maintenance and adjustment may be performed, according to the manufacturer's standard procedure.

When all tests, inspections, repairs, and adjustments have been completed, normal operation shall be verified by conducting an operational status check.

During this process, all equipment shall be operated in a manner and environmental conditions that simulate election use to verify the functional status of the system.  Prior to the conduct of each of the environmental hardware non-operating tests, a supplemental test shall be made to determine that the operational state of the equipment is within acceptable performance limits.

The following procedures shall be followed to verify the equipment status:

Step 1:     Arrange the system for normal operation.

Step 2:     Turn on power, and allow the system to reach recommended operating temperature.

Step 3:     Perform any servicing, and make any adjustments necessary, to achieve operational status.

Step 4:     Operate the equipment in all modes, demonstrating all functions and features that would be used during election operations.

Step 5:     Verify that all system functions have been correctly executed.

Upon completion of each non-operating test, the system hardware shall be subject to functional testing to verify continued operability. If any portion of the voting machine or precinct counter hardware fails to remain fully functional, the testing will be suspended until the failure is identified and corrected by the vendor. The system will then be subject to a retest.

The bench handling test simulates stresses faced during transport and handling of voting machines and ballot counters.

All systems and components, regardless of type, shall meet the requirements of this test.  This test is equivalent to the procedure of MIL-STD-810D, Method 516.3, Procedure VI.

Step 1:     Place each piece of equipment on a level floor or table, as for normal operation or servicing.

Step 2:     Make provision, if necessary, to restrain lateral movement of the equipment or its supports at one edge of the device.  Vertical rotation about that edge shall not be restrained.

Step 3:     Using that edge as a pivot, raise the opposite edge to an angle of 45 degrees, to a height of four inches above the surface, or until the point of balance has been reached, whichever occurs first.

Step 4:     Release the elevated edge so that it may drop to the test surface without restraint.

Step 5:     Repeat steps 3 and 4 for a total of six events.

Step 6:     Repeat steps 2, 3, and 4 for the other base edges, for a total of 24 drops for each device.

Step X:    Remove the test item from its transit or combination case and verify its continued operability.

The vibration test simulates stresses faced during transport and handling of voting machines and ballot counters.

All systems and components, regardless of type, shall meet the requirements of this test.  This test is equivalent to the procedure of MIL-STD-810D, Method 514.3, Category 1- Basic Transportation, Common Carrier.

Step 1:     Install the test item in its transit or combination case as prepared for delivery to the polling site.

Step 2:     Attach instrumentation as required to measure the applied excitation.

Step 3:     Mount the equipment on a vibration table with the axis of excitation along the vertical axis of the equipment.

Step 4:     Apply excitation as shown in MIL-STD-810D, Method 514.3-1, “Basic transportation, common carrier, vertical axis”, with low frequency excitation cutoff at 10 Hz, for a period of 30 minutes.

Step 5:     Repeat steps 2 and 3 for the transverse and longitudinal axes of the equipment with the excitation profiles shown in Figures 514.3-2 and 514.3-3, respectively. (Note:  The total excitation period equals 90 minutes, with 30 minutes excitation along each axis.)

Step X:    Remove the test item from its transit or combination case and verify its continued operability.

The low temperature test simulates stresses faced during transport and storage of voting machines and ballot counters.

All systems and components, regardless of type, shall meet the requirements of this test.  This test is equivalent to the procedure of MIL-STD-810D, Method 502.2, Procedure I-Storage.  The minimum temperature shall be -4 degrees F.

Step 1:     Arrange the equipment as for storage.  Install it in the test chamber.

Step 2:     Lower the internal temperature of the chamber at any convenient rate, but not so rapidly as to cause condensation in the chamber, and in any case no more rapidly than 10 degrees F per minute, until an internal temperature of -4 degrees F has been reached.

Step 3:     Allow the chamber temperature to stabilize.  Maintain this temperature for a period of 4 hours after stabilization.

Step 4:     Allow the internal temperature of the chamber to return to standard labo­ratory conditions, at a rate not exceeding 10 degrees F per minute

Step 5:     Allow the internal temperature of the equipment to stabilize at laboratory conditions before removing it from the chamber.

Step 6:     Remove the equipment from the chamber and from its containers, and inspect the equipment for evidence of damage.

Step 7:     Verify continued operability of the equipment.

The high temperature test simulates stresses faced during transport and storage of voting machines and ballot counters.

All systems and components, regardless of type, shall meet the requirements of this test.  This test is equivalent to the procedure of MIL-STD-810D, Method 501.2, Procedure I-Storage.  The maximum temperature shall be 140 degrees F.

Step 1:     Arrange the equipment as for storage.  Install it in the test chamber.

Step 2:     Raise the internal temperature of the chamber at any convenient rate, but in any case no more rapidly than 10 degrees F per minute, until an internal temperature of 140 degrees F has been reached.

Step 3:     Allow the chamber temperature to stabilize.  Maintain this temperature for a period of 4 hours after stabilization.

Step 4:     Allow the internal temperature of the chamber to return to standard labo­ratory conditions, at a rate not exceeding 10 degrees F per minute.

Step 5:     Allow the internal temperature of the equipment to stabilize at laboratory conditions before removing it from the chamber.

Step 6:     Remove the equipment from the chamber and from its containers, and inspect the equipment for evidence of damage.

Step 7:     Verify continued operability of the equipment.

The high temperature test simulates stresses faced during storage of voting machines and ballot counters.

All systems and components regardless of type shall meet the requirements of this test.  This test is similar to the procedure of MIL-STD-810D, Method 507.2, Procedure I-Natural Hot-Humid.  It is intended to evaluate the ability of the equipment to survive exposure to an uncontrolled temperature and humidity environment during storage.  This test lasts for ten days.

The equipment shall be in a non-operating, storage configuration, and a protective cover or enclosure shall be in place if one is intended to be used during storage.

Step 1:     Install the equipment in the test chamber.  Adjust the chamber conditions to those given in MIL-STD-810D Table 507.2-I, for the time 0000 of the Hot­Humid cycle (Cycle 1).

Step 2:     Perform a 24-hour cycle with the time and temperature-humidity values specified in Figure 507.2-1, Cycle 1.

Step 3:     Repeat Step 2 until 5, 24-hour cycles have been completed.

Step 4:     Continue with the test commencing with the conditions specified for time = 0000 hours.

Step 5:     At any convenient time in the interval between time = 120 hours and time = 124 hours, place the equipment in an operational configuration, and perform a complete operational status check as defined in Subsection          4.6.1.5

Step 6:     If the equipment satisfactorily completes the status check, continue with the sixth 24-hour cycle.

Step 7:     Perform 4 additional 24-hour cycles, terminating the test at time = 240 hours

Step 8:     Remove the equipment from the test chamber and inspect it for any evidence of damage.

Step 7:     Verify continued operability of the equipment.

This section addresses a range of tests for all voting system equipment, including equipment for both precinct count and central count systems.

This test is similar to the low temperature and high temperature tests of MIL-STD­810D, Method 502.2 and Method 501.2, with test conditions that correspond to the requirements of the performance standards.  The temperature range for equipment operation shall be:

          Ambient Temperature Range, degrees F:  

                 Minimum = 50

                 Maximum = 95

In this test, the software need only operate to the extent necessary to enable the identification of hardware failures or the suspected inability of the system to perform all of the functions to be evaluated in the Functional Configuration Audit during system-level testing.  COTS hardware, as defined previously, may not be subjected to the 48-hour chamber segment of the operating environmental tests.

This procedure tests system operation under various environmental conditions for at least 163 hours. During 48 hours of this operating time, the device shall be in a test chamber.  For the remaining hours, the equipment shall be operated at room temperature.  The system shall be powered for the entire period of this test; the power may be disconnected only if necessary for removal of the system from the test chamber.

Operation shall consist of ballot-counting cycles, which vary with system type.  An output report need not be generated after each counting cycle; the interval between reports, however, should be no more than 4 hours to keep to a practical minimum the time between the occurrence of a failure or data error and its detection.

          Test Ballots per Counting Cycle

                 Precinct count systems      100 ballots/hour

                 Central count systems       300 ballots/hour

The recommended pattern of votes is one chosen to facilitate visual recognition of the reported totals; this pattern shall exercise all possible voting locations.  System features such as data quality tests, error logging, and audit reports shall be enabled during the test.

Each operating cycle shall consist of processing the number of ballots indicated in the preceding chart. 

Step 1:     Arrange the equipment in the test chamber.  Connect as required and provide for power, control and data service through enclosure wall.

Step 2:     Set the supply voltage at 117 vac.

Step 3:     Power the equipment, and perform an operational status check as in Sec­tion 4.6.1.5.

Step 4:     Set the chamber temperature to 50 degrees F observing precautions against thermal shock and con­densation.

Step 5:     Begin 24 hour cycle.

Step 6:     At T=4 hrs, lower the supply voltage to 105 vac.

Step 7:     At T=8 hrs, raise the supply voltage to 129 vac.

Step 8:     At T=11:30 hrs, return the supply voltage to 117 vac and return the chamber temperature to lab ambient, observing precautions against thermal shock and condensation.

Step 9:     At T=12:00 hrs, raise the chamber temperature to 95 degrees Fahrenheit.

Step 10:   Repeat Steps 5 through 8, with temperature at 95 degrees Fahrenheit, complete at T=24 hrs.

Step 11:   Set the chamber temperature at 50 degrees Fahrenheit as in Step 4.

Step 12:   Repeat the 24 hour cycle as in Steps 5-10, complete at T=48 hrs.

Step 13:   After completing the second 24 hour cycle, disconnect power from the system and remove it from the chamber if needed.

Step 14:   Reconnect the system as in Step 2, and continue testing for the remaining period of operating time required until the ACCEPT/REJECT criteria of Subsection 4.7.4 have been met.

The test for power disturbance disruption shall be conducted in compliance with the test specified in in IEC 61000-4-11 (1994-06).

The test for electromagnetic radiation shall be conducted in compliance with the FCC Part 15 Class B requirements by testing per ANSI C63.4.

The test for electrostatic disruption shall be conducted in compliance with the test specified in IEC 61000-4-2 (1995-01).

The test for electromagnetic susceptibility shall be conducted in compliance with the test specified in IEC 61000-4-3 (1996).

The test for electrical fast transient protection shall be conducted in compliance with the test specified in IEC 61000-4-4 (1995-01).

The test for lightening surge protection shall be conducted in compliance with the test specified in IEC 61000-4-5 (1995-02).

The test for conducted RF immunity shall be conducted in compliance with the test specified in IEC 61000-4-6 (1996-04).

The test for AC magnetic fields RF immunity shall be conducted in compliance with the test specified in IEC 61000-4-8 (1993-06).

Successful completion of the Operating Environmental tests shall be determined by two criteria:

a.       The number of system failures: A system shall be rejected if it fails to pass the operating environmental test.

b.       Accuracy of vote data: The accuracy of the vote count evaluated using the test design and procedures described in Appendix C.  Subsection C.6 contains step by step protocols for resolving discrepancies during data accuracy testing.

As indicated in Volume I, Section 3, data accuracy is defined in terms of ballot position error rate. This rate applies to the voting functions and supporting equipment that capture, record, store, consolidate and report the specific selections, and absence of selections, made by the voter for each ballot position. Volume I, Section 3.2.1 identifies the specific functions to be tested.

For each processing function, the system shall achieve a target error rate of no more than one in 10,000,000 ballot positions, with a maximum acceptable error rate in the test process of one in 500,000 ballot positions. This error rate includes errors from any source while testing a specific processing function and it related equipment.

This error rate is used to determine the vote position processing volume used to test system accuracy for each function:

¨       If the system makes one error before counting 26,997 consecutive ballot positions correctly, it will be rejected. The vendor is then required to improve the system.

¨       If the system reads at least 1,549,703 consecutive ballot positions correctly, it will be accepted.

¨       If the system correctly reads more than 26,997 ballot positions but less than 1,549,703 when the first error occurs, the testing will have to be continued until another 1,576,701 consecutive ballot positions are counted without error (a total of 3,126,404 with one error).

Volume II, Appendix C, Section C.5 provides further details of the calculation for this testing volume.

The ITA shall test for maintainability based on the provisions of Volume I, Section 3 for maintainability, including both physical attributes and additional attributes regarding the ease of performing maintenance activities. These tests include:

a.      Examine the physical attributes of the system to determine whether significant impediments exist for the performance of those maintenance activities that are to be performed by the jurisdiction. These activities shall be identified by the vendor in the system maintenance procedures (part of the TDP).

b.      Performing activities designated as maintenance activities for the jurisdiction in the TDP, in accordance with the instructions provided by the vendor in the system maintenance procedures, noting any difficulties encountered.

Should significant impediments or difficulties be encountered that are not remedied by the vendor, the ITA shall include such findings in the qualification test results of the qualification test report.

The ITA shall test for reliability based on the provisions of Volume I, Section 3 for the acceptable mean time between failure (MBTF). The MBTF shall be measured during the conduct of other system performance tests specified in this section, and shall be at least 163 hours. Volume II, Appendix C, Section C.4 provides further details of the calculation for this testing period.

The ITA shall assess the adequacy of system availability based on the provisions of Volume I, Section 3. As described in this section, availability of voting system equipment is determined as a function of reliability, and the mean time to repair the system in the event of failure.

Availability cannot be tested directly before the voting system is deployed in jurisdictions, but can be modeled mathematically to predict availability for a defined system configuration. This model shall be prepared by the vendor, and shall be validated by the ITA.

The model shall reflect the equipment used for a typical system configuration to perform the following system functions:

a.       For all paper-based systems:

1)      Recording voter selections (such as by ballot marking or punch);

2)      Scanning the punches or marks on paper ballots and converting them into digital data;

b.       For all DRE systems:

1)      Recording and storing the voter’s ballot selections.

c.       For precinct-count systems (paper-based and DRE):

1)      Consolidation of vote selection data from multiple precinct-based systems to generate jurisdiction-wide vote counts, including storage and reporting of the consolidated vote data; and

d.       For central-count systems (paper-based and DRE):

1)      Consolidation of vote selection data from multiple counting devices to generate jurisdiction-wide vote counts, including storage and reporting of the consolidated vote data.

The model shall demonstrate the predicted availability of the equipment that supports each function. This demonstration shall reflect the equipment reliability, mean time to repair and assumptions concerning equipment availability and deployment of maintenance personnel stated by the vendor in the TDP.