Talking Notes for Programme Performance Indicators (PPI)
Meeting (21 May 2015)
Background
The current performance indicators have been in place for a number of years:
VEA targets being set in 1998-1999; and
SRCA and MRCA targets being set in 2005-2006.
Late last year, the SES forum in Canberra agreed that DVA’s current PPIs were no
longer appropriate.
The PGPA set out assessment criteria to be followed in reviewing the PPIs.
In reviewing the performance indicators against this criteria, it has become obvious
the
programme objectives and deliverables (under Outcome 1) were inconsistent,
some were irrelevant, and others either not accurate or missing.
In summary, the 3 key points are:
1. The objectives and deliverables of Outcome 1 do not accurately reflect their
respective programmes;
2. Measurement of timeliness on the basis of the mean is of limited value and is out
of step with the direction pursued by other agencies with comparable business;
and
3. The assessment of quality currently emphasises the negative and does not
clearly exclude those factors that have no impact on the outcome for the client.
Issues
From the EMB feedback received, it would appear the issue for Outcome 1 is about
managing negative perceptions and reactions of stakeholders (Veterans, ESOs and
the Senate) if a decision is made to change the timeliness indicators for R&C VEA
claims from 75 days to 120 days for VEA Disability claims and to 30 days for VEA
War/Widow/ers claims. In the context of this potential challenge, I will concentrate
my points on the changes to programme 1.2 and 1.3.
Timeliness indicator
The current timeliness indicator is the “mean number of days” to process a claim.
The weakness of this measure is that it is easily misunderstood, with the veteran
community often overlooking that it is an average. It is also easily distorted by
finalisation or merging of very old or very young cases.
A review of other Departments’ timeliness targets shows that most use a percentile
band as a timeliness measure. The strength of the percentile measure is that it is
intrinsically understandable, shows what will happen with a typical case and is not
distorted by a few very old or very young cases.
1
Current R&C performance data for 2014-15 FYTD, would suggest that the
introduction of percentile measure is still some way off. However, the 50th percentile
(also known as “the median”) would be a positive starting point and a good
introduction to percentile measures.
The use of the median to assess TTTP was also a recommendation of the R&C
Improvement Project.
Table 1 shows R&C targets and 2014-15 FYTD mean as well as timeliness results
against the median and percentile point at the current target.
Table 1 – VEA Targets and 2014-15 FYTD (1 July 2014- 17 may 2015 )Performance
Claims
Finalised
Percentile
Target
Mean
Median
Programme
(including
point at
Days
Days
Days
merged and
Target
withdrawn)
1.2 VEA Primary
11,688
Claims (currently
75
72
54
61th
incorrectly included
WW claims)
1.2 Disability claims
10,183
(DP,AFIS & DP
Assessment)-
75
76
59
58th
excluding war
widow/ers claims
1.3 War Widow/ers
1,505
75
44
24
82th
only
The data currently used to provide the timeliness figures is incorrect in programme
1.2 and 1.3. This data is based on all VEA Compensation Primary Claims, however
1.2 is only related to disability pensions and allowances and 1.3 is specifically related
to widow/ers and dependants pensions.
In addition, when a client lodges both a disability pension claim and an application
for increase (AFI) under the VEA, both claims are registered. For ease of case
management and to streamline disability pension assessment and advices upon
finalisation, the AFI is generally registered and then immediately “merged”. This
shows that two separate claims were lodged, but leaves only one case outstanding.
Although such AFIs are actually only completed together with the disability pension
claim with which they have been merged, these merged AFIs count as finalised
cases for the purposes of VEA timeliness. The number of these cases is not
insignificant, amounting to almost 30 per cent of cases finalised each year, and all
with a TTTP on only one day. The effect of this practice on mean TTTP is clearly
shown in the tables below.
2
Table 2 shows the effect of including and excluding merged and withdrawn cases on
VEA mean TTTP for 2014-15 FYTD.
Table 2 – VEA Mean TTTP 2014-15 FYTD (1 July 2014- 17 may 2015 )
Disposals including
Disposals
Finalised
Claim Type
Finalised, Merged and
cases only, Merged and
Withdrawn cases
Withdrawn cases excluded
VEA Primary Claims (currently
72
94
incorrectly used in 1.2)
Disability claims (DP, AFIS & DP
Assessment) – excluding war
76
102
widow/ers claims ( correct data for
1.2)
1.3 War Widow/ers only ( correct
44
46
data for 1.3)
Table 3 shows the effect of including and excluding merged and withdrawn cases on
VEA median TTTP for 2014-15 FYTD.
Table 3 – VEA Median TTTP 2014-15 FYTD (1 July 2014- 17 may 2015 )
Disposals including
Disposals
Finalised
Claim Type
Finalised, Merged and
cases only, Merged and
Withdrawn cases
Withdrawn cases excluded
VEA Primary Claims (currently
54
78
incorrectly used in 1.2)
Disability claims (DP, AFIS & DP
Assessment) – excluding war
59
88
widow/ers claims (correct data for
1.2)
War Widow/ers only (correct data
24
27
for 1.3)
It is clear that this way of measuring performance is an inaccurate indicator, with the
inclusion of merged cases essentially distorting the timeliness results. This should
be amended to more accurately reflect DVA’s performance.
Not addressing this now may have significant consequences for the timeliness
results over time, if the proportion of cases that are both AFIs and disability claims
decreases. It should be understood, however, that addressing this issue will have
the immediate effect of increasing reported timeliness. This should be offset by
explaining that the disability claims now exclude war widow/ers claims. Furthermore,
the revised target for all initial liability claims (under VEA, SRCA and MRCA) would
be 120 days.
3
The War Widow/ers results clearly indicate that the target could easily be reduced to
30 days instead of 75 days, if the median was used.
Quality indicator
Currently the quality indicator is the critical error rate.
This measure is also easily misunderstood, with the veteran community having little
understanding of what constitutes a critical error. For example, it could imply that 1
in 20 payments/pensions are incorrect and this error rate is acceptable to DVA.
The data used to provide the quality results currently captures a variety of errors
whereas other agencies like the Department of Human Service (DHS) clearly
distinguishes between financial impact errors and administrative errors.
A review of Australian National Audit Office (ANAO) audits on performance
measures used by DHS in relation to payment integrity highlighted that they used
two terms, ‘payment accuracy’ and ‘payment correctness’ when reporting quality
results against their programmes.
The ANAO recommended that DHS make transparent, in its Annual Report (and any
other documents where the agency reports on its level of payment accuracy and
payment correctness), how the figures are derived to allow readers to properly
interpret and understand the data and have confidence in the results.
In terms of payment correctness, a payment is considered ‘correct’ if: the right
person is paid; under the right programme; at the right rate; for the right date(s).
‘Correctness’ relates only to decision-making processes within DHS’s control.
DVA could use the term ‘Correctness’ as it relates only to decision-making
processes within the Department’s control. This would be consistent with
comparable agencies and allow for comparison of performance.
Price indicator
The ‘price’ indicator is only listed under programme 1.1 and 1.2 and as a standalone
measure, does not offer any meaningful information about the programme. This
indicator was removed.
Recommendations
The PBS objectives, deliverables, and performance indicators should be reviewed
annually to ensure they remain relevant and appropriate.
Timeliness Indicator
Programme 1.2
The timeliness data should:
4
remove war widow/ers claims; and
exclude merged and withdrawn cases.
The timeliness indicator should be:
based on percentile bands, commencing with the median in 2015-16; and
changed to 120 days.
These recommendations would reflect more accurately, the time it takes for a
Disability claim to be determined. It also aligns with initial liability claims under MRCA
and SRCA, so instead of reporting our timeliness by Act we would be reporting by
type of claim (i.e. Liability claim).
Once improvements in timeliness are proven to be sustainable then the median as a
measure should be removed and a higher percentile point be introduced.
Programme 1.3
The timeliness data should only include war widow/ers claims.
The timeliness indicator should be changed to 30 days.
Quality Indicator
The ‘quality’ indicator should be rephrased to ‘correctness rate’ or ‘accuracy rate’
and include a clearly articulated explanation of how the rate is derived to ensure
readers are able to easily and properly interpret the results.
The quality data should only represent factors that impact on the programme
outcomes, i.e. direct financial impact to the client and exclude administrative errors.
Price Indicator
The ‘price’ indicator is removed from programmes 1.1 and 1.2.
Validation
From meetings with Barry Press it is my understanding that the suggested changes
to the indicators were validated against the assessment criteria and received positive
support.
Maralyn Newman
20 May 2015
5
