Relevant Document.pdf

Test Summary Report
for the
Counting the Senate Project
Prepared by
IV&V Australia Pty Ltd
Prepared for
Australian Electoral Commission
Date: 10 July 2018
IVVAUST/AEC/086-01
Accredited for compliance with ISO/IEC 17025.
Authorised by
Name
Donna O’Neill
IV&V Australia Pty Ltd ABN 13 073 484 287
Function
Managing Director
Suite 107, 6-8 Clarke Street
Crows Nest, NSW 2065 Australia
Signature

Contents

1.
Introduction .................................................................................................................... 3
1.1  Identification .......................................................................................................................... 3
1.2  Product Description .............................................................................................................. 3
1.3  Test Execution Summary ...................................................................................................... 3
1.4  Referenced Documents ......................................................................................................... 4
2.
Test Execution Discussion ............................................................................................ 5
2.1  Summary of Test Results ...................................................................................................... 5
2.2  Summary of the Testing Performed ................................................................................... 5
2.2.1
The Test Approach .......................................................................................................................... 5
2.2.2
Test Method/Test Case Selection .................................................................................................. 6
2.2.3
Test Harness Validation ................................................................................................................. 7
2.2.4
Formal Test Activities ..................................................................................................................... 8
2.3  Conformance with Protocol ................................................................................................. 9
2.4  Limitations with the Test Process .................................................................................... 10
3.
Detailed Test Results ................................................................................................... 11
4.
Summary of Outstanding Issues ............................................................................... 13
5.
Test Environment ........................................................................................................ 14

•  Conduct dry run testing using the AEC-supplied tests to review the test procedures and
to prepare for the formal NATA test run.
•  Conduct the formal NATA test.
•  Prepare a NATA-endorsed Test Summary Report (this document).
2.2.2  Test Method/Test Case Selection
AEC supplied the Count and BPRS test cases, spreadsheets and data files used during testing.
IV&V Australia conducted a desk review of the AEC test strategy, test cases and test data to
determine the adequacy of the test baseline.  Specifically, the review confirmed whether:
•  The data spreadsheets correctly reflect changes to the algorithm;
•  The test steps correctly reflect the modified spreadsheets;
•  The test cases adequately cover the changes made;
•  The intents of the test cases were still met, given the changes;
•  The test cases that are considered to be out of scope by the AEC should be included in
IV&V’s test set; and
•  The BPRS test cases adequately provide assurance over the integration of data into ECS.
This review was conducted using the test cases and test data spreadsheets

as the primary reference for the ECS modifications.

were provided to the AEC.
The test case review was followed by a test procedure review/informal dry run of the tests to
check test step flow, completeness of prerequisites, adequacy of datasets and adequacy of
the test environment.
were provided to the AEC.
Both reviews helped to identify potential improvements to the test cases and test procedures,
such as:
•  Minor typographical errors that should be ignored
•  Additional free-play test steps and/or test data to improve test coverage
•  A more efficient way of ordering and running the tests.
During testing, this took the form of test step deviations, document “mark-ups” and free-
played test steps.  These changes are described in detail in Section 2.3 Conformance with
Protocol.
The test case review identified that the test data for one test case (2.2.9 Surplus distribution in
a different order to election) that has been previously included in the Count test suite was not
supplied by the AEC. AEC advised that this is an area of low risk and so could be omitted.
The test procedure review identified that the test steps for validating SSO login were weak.
AEC advised that this functionality should be removed from the scope of testing.
The reviews also raised some questions about the use of the Load Test Scenario functionality/
test harness. AEC provided a test harness validation procedure to help mitigate any concerns.
This is discussed in Section 2.2.3 below.

2.2.3  Test Harness Validation
Load Test Scenario
Most of the Count test suite consists of small artificial electoral events which are designed to
provoke specific scenarios.  The AEC provided a test harness for loading the ballot data for
these artificial electoral events into ECS.  This is a combination of the Load Test Scenario
functionality in ECS and a SQL script for clearing the database between electoral events.
This data loading process is different to the mechanism to be used in production and different
to the method used for the most recent few rounds of ECS NATA testing.  This raised
questions about whether a count based on data loaded by the test harness would always give
the same result as data loaded via BPRS in production.
To examine this, IV&V Australia conducted a test harness validation activity, as follows:
•  Executed an AEC-supplied test harness validation procedure, TC 64106.  TC 64106
demonstrates that the developed Load Test Scenario data importer tool and the BPRS
VCP Import Data function produces the same notional preferences, count results, and
Distribution of Preferences report.  This was demonstrated using 2016 ACT Federal
Senate election data.
•  Reviewed and free-played the associated SQL Clear script plus the two stored
procedures it invokes, to show that it correctly reinitialised the database and did not
appear to introduce any unintended side effects
There were no test step failures while running TC 64106 or in the SQL script review and there
were no differences in the numbers in the reports; however, there were minor differences in
the annotations in the reports.
It should be noted that there are differences between ECS using an electoral event loaded
using Load Test Scenario and ECS using an electoral event loaded via BPRS.  Whilst the final
Count result appears to be identical, some minor uncertainty remains because:
•  The two methods use different software to convert from the actual ballot data to the
“Notional BTL Preferences” which raises the possibility that a software error in this area
might pass the tests but introduce a Count error in production.
•  Load Test Scenario cannot load informal ballot papers.  This is a minor issue, as none of
these artificial test scenarios included any informal ballot papers.
•  The converted ballots are presented as coming from a single VCP, as opposed to the
data loaded through BPRS that comes from multiple VCPs. As a consequence, ECS
reports that the data loaded through Load Test Scenario leaves Unverified VCPs and
that these are not being included in the count.
It should be noted that there is a minor weakness in the test harness validation method in
that the conversion of the actual 2016 ACT Federal Senate ballots to the form required to
load via Load Test Scenario is not itself validated.
SQL Compare Script
SQL Compare script was written to compare the ballot papers and count results in two
databases.  Typically, this was used to compare the results of the 2016 Federal Senate events

(counted using the version of ECS current at that time) with the results when the data was
loaded through BPRS and counted with the new version of ECS.
The validation consisted of free-play testing, and the conclusion was that the script appears
to correctly compare two ECS databases for:  Ballot Paper Counts, Formal Ballot Paper
Counts, Notional Preferences, Results, and Audit Trails.

2.2.4  Formal Test Activities
The formal NATA test event involved the following test activities:
•  Test execution was done in two phases - a data capture phase followed by an analysis
phase. The start and stop times for each phase were documented on the test logs for
each test case.
•  The data capture phase was conducted at AEC’s Sydney premises on a combination of
the DEV and PROD environments.  Test environment information was captured in a
number of screenshots and configuration files, which are discussed in Section 5 of this
document.

•  The analysis phase was conducted at IV&V Australia’s premises.
•  The formal test executed a library of AEC-supplied Count and BPRS test cases,
spreadsheets and data files.

•  All of the test cases were run on ECS software version 4.0.35.51606, and where
relevant, BPRS software version 2.1.50.51128.
•  All of the test cases required some adaptation and deviations during test execution.
These were identified on the test log for the test case and are discussed in Section 2.3
of this document.
•  ESRP checks (regression tests). During the 2016 NATA testing for the Senate Reform
Project, three defects remained Open/Resolved at the completion of testing:
•  For this current version of ECS under test, ESRP-97 was again observed during
normal testing but it is considered to be Resolved and did not cause a failure.
•  AEC advised that ESRP-84 and ESRP-95 cannot occur in the ECS version under test.
IV&V conducted some free-play exploration around this functionality to create
conditions as close as reasonably practical to those conditions.  The issues were not
observed and so it can reasonably be assumed that they have been fixed.
•  All test cases Passed, with no failed steps.
•  There were two issues reported to AEC. One was resolved by skipping an invalid test
step and the other was a cosmetic issue with the Distribution of Preferences Report.
•  The Pass/Fail test results for each test case are provided in Section 3 of this document.

•
  These
include:
•  The scanned test cases with pass/fail results and test step mark-ups
•  The test logs for each test case, with free-play steps documented
•  The reports generated during the data capture phase
•  ESRP free-play test checks
•  Test harness validation records
•  Correspondence on issues seen during testing.
The testing was conducted in accordance with IV&V Australia’s testing process, which is
accredited by the National Association of Testing Authorities, Australia (NATA) to be in
compliance with ISO 17025-2005 General requirements for the competence of testing and
calibration laboratories.
2.3  Conformance with Protocol
Testing was performed in conformance with the
with the
following exceptions:
•  Two test execution phases.  Most tests were conducted in two phases.  During the first
phase, sufficient test steps were executed such that all of the reports required by any of
the test cases were generated as a data capture activity. The second phase consisted of
a subsequent analysis of the reports and verification of the results using those reports.
•  Test cases that had common setup steps.  Some Count test cases have similar setup
steps to load the data file and run a count, with the differences being solely in the
analysis of the reports generated.
The setup steps were confirmed to be correct for each test case, but the steps were not
actually re-run.  The test cases were grouped together on one test log, with an
indication that the setup was performed at the start of the sequence.
•  Test cases that required adaptation to flow procedurally.  Some test cases had test
steps which did not flow as written and required some adaptation during test execution
to achieve the expected results.
Where these work-arounds occurred, the deviations were noted in the test logs.  Care
was taken to ensure that the intent of the tests was still met.
•  Test cases that required additional verification steps to improve test coverage (eg,
generating and analysing additional reports).  These steps were executed as free-play
steps and documented in the test logs.
•  Test steps that were not run.  TC 57039 had an incorrect test step that was skipped and
test steps that relate to SSO were skipped.
•  Test cases that were duplicates of each other.  Some Count tests are essentially
identical to each other, with only trivial differences in documentation detail. The test
cases were grouped together on one test log, with an indication that as the first test
passed, the others passed also.

2.4  Limitations with the Test Process
During the pre-formal test activities (ie, reviews, dry run), IV&V Australia raised a number of
concerns regarding technical limitations and constraints with the test process and the test
methods available for IV&V to use (as opposed to those used for previous projects).
The more critical elements of these were largely mitigated prior to the formal test. To
summarise:
•  Limitations in test coverage.  The test case review found that there were a number of
specific verifications removed from the Count test case suite (from the previous NATA
testing), which weakened the suite.  For the most part, IV&V Australia effectively
remediated these limitations by free-playing extra steps during the tests to reinstate the
verifications (using the previous tests as a guide).  One previous test case (2.2.9) had to
be omitted entirely because the test data was not available in the test environment.
With regards the BPRS test cases, IV&V’s scope of work was largely to demonstrate that
it is possible to import data through BPRS to ECS and have it counted as expected.  The
wider BPRS functionality was out of scope and IV&V did not have visibility into any other
testing of this functionality (eg, negative tests).
•  Differences to the production environment.  Previous NATA testing of ECS has been done
in environments that were set up by the AEC to be as close as practical to the real
production environment in which ECS would run. For Counting the Senate testing, there
were differences to the production environment which raised some concerns.
Whilst each of these differences is individually low risk, taken together they created
uncertainty as to whether the test results are applicable to the final software run in a
properly configured production environment.
•  Data loading. The new method of data loading is different to how it will be in
production and different to what has been done for the last few rounds of IV&V’s
ECS testing. The differences add minor potential for concealing a defect (see Section
2.2.3 Test Harness Validation).
•  Test environment. This round of testing has involved some components running on
“office” production PCs and on a PC in a DEV environment.  The DEV environment
had some non-production software loaded (eg SQL Server Management Studio).  This
raises the potential for defects that exist in the production environment to be missed
when using the test configuration.