Program evaluation

Page last updated: 24 October 2014

The final report of the BreastScreen Australia Evaluation Program was released on 7 September 2009. The report’s methodology, key findings and recommendations are outlined in the Executive Summary. The complete report and the formal response from the Australian Ministers Health Advisory Council (AHMAC) are available from the Australian Government Web Archive.

BreastScreen Australia Evaluation Final Report

Executive Summary

Evaluation Recommendations

Recommendations


    1. Continue BreastScreen Australia as a population-based screening program, providing biennial breast cancer screening using mammography for women in the target age group.

    2. Use evidence of the greatest mortality benefit to determine a nationally agreed target age group for the Program. Based on the evidence, preference should be given in the following order to:

      i. increasing the participation rate of women aged 50–69 years;

      ii. extending the target age range to provide biennial screening for women aged 70– 74 years;

      iii. extending the target age range to provide annual screening for women aged 45–49 years;

      iv. extending the target age range to provide biennial screening for women aged 45–49 years.

    3. Women aged 40–44 years and women aged 75 years and over should no longer be eligible to attend the Program.

    4. Develop clear national policies for BreastScreen Australia that address inconsistent policy application across jurisdictions to ensure equity for women.


Increasing participation


    5. Focus on increasing the screening participation rate of the target age group to maximise mortality reduction through a range of social marketing, communication and service delivery strategies, while ensuring there is sufficient capacity to meet demand.

Sub-recommendations
      - Increase the screening participation rate of Aboriginal and Torres Strait Islander women, women from culturally and linguistically diverse backgrounds, women living in very remote areas and women living in major cities.

      - Increase rescreening rates in the target population, with a particular emphasis on increasing the rescreening rate between the first screening visit and subsequent rescreening.

      - Promote the results of the Evaluation to women and health care providers to improve recognition of BreastScreen Australia and increase women’s understanding of breast cancer screening, including its benefits and potential harms.

Governance and management


    6. Improve national policy leadership and Program development to facilitate timely decision making, direction setting and capacity to deal with emerging issues and new technology.

    7. Review jurisdictional governance and management to ensure optimal capacity to deliver an agreed service model and achieve policy consistency and operational efficiency.



Program capacity and infrastructure


    8. Address capacity constraints on Program effectiveness through implementing enhanced workforce practices and service models and maximising the benefits of digital mammography.

    9. Focus on minimising delays between screening and assessment of women with a screen-detected abnormality.



Screening policy in relation to breast cancer risk


    10. Women at a potentially high risk of breast cancer (i.e. a relative risk greater than three times population risk) should not be screened within BreastScreen Australia due to their need for individualised assessment including more intensive surveillance and monitoring. National protocols should be implemented consistently for managing women at high risk of breast cancer who present to BreastScreen Australia services, including appropriate referral. This includes:

      - women at potentially high risk due to a strong family history of breast cancer, as defined by National Breast and Ovarian Cancer Centre guidelines;

      - women with identified high-risk genetic mutations, such as BRCA1 or BRCA2; and

      - women with a recent (<5 years) diagnosis of invasive breast cancer or ductal carcinoma in situ.


    11. Implement national policies to ensure women are screened at the appropriate interval according to their level of risk. Family history risk should be assessed using the National Breast and Ovarian Cancer Centre family history tool.

Sub-recommendations
      - Women with a previous history of invasive breast cancer or ductal carcinoma in situ: BreastScreen Australia should provide access to annual screening to women from 5 years following diagnosis. Apart from mammography, ongoing surveillance of these women should be provided by their usual health care provider.

      - Women with a history of atypical hyperplasia (atypical ductal hyperplasia or atypical lobular hyperplasia) or lobular carcinoma in situ: BreastScreen Australia should provide access to annual screening to women for at least 15 years following diagnosis. Apart from mammography, ongoing surveillance of these women should be provided by their usual health care provider.

      - Women at moderate risk of breast cancer due to family history: BreastScreen Australia should provide access to biennial screening to women.


    12. Women with symptoms: BreastScreen Australia should not include women with symptoms in the Program. Women with symptoms require individualised assessment using the triple test approach in accordance with National Breast and Ovarian Cancer Centre guidelines. National protocols should be implemented consistently for managing women who present with symptoms to BreastScreen Australia services, including appropriate referral. Recommendation 19 also refers.

    13. Develop a national policy based on a review of the available evidence in relation to identifying and managing women according to their breast density, following the implementation of digital mammography in BreastScreen Australia.



Research and data


    14. Improve the quality, accessibility, consistency and timeliness of reporting of nationally agreed standard data items to address current inadequacies of data collection and provision.

    15. Use Program data to:


      - inform policy development;

      - monitor and evaluate Program performance;

      - review Program performance indicators;

      - enable strategic research; and

      - enhance opportunities for learning across the Program, including through sharing accreditation performance data.




Quality improvement


    16. Strengthen the accreditation system, within a broader quality improvement framework, by modifying the role, membership and operation of the National Quality Management Committee. Consideration should be given to:

      - separating the standard-setting responsibilities from the accreditation assessment processes;

      - providing an independent chair;

      - including more members who are independent of Program delivery; and

      - ensuring greater transparency.


    17. Refine the national accreditation process to reduce its overall burden on services and site visitors.

    18. Refine both the number and structure of the national accreditation standards to focus on standards that align more closely with the objectives of the Program, and establish ongoing mechanisms for review of the standards.



Exploring alternative service models


    19. Explore the potential benefits and feasibility of co-location or greater integration of screening and diagnostic services through pilot or demonstration studies. Possible areas for exploration are:

      - diagnostic assessment of women with symptoms of breast cancer;

      - individualised surveillance of women at potentially higher risk of breast cancer due to family history;

      - individualised surveillance post-treatment of women with a recent (<5 years) invasive breast cancer or ductal carcinoma in situ; and

      - individualised risk assessment of women of any age to assess their risk for breast cancer and recommend appropriate screening or surveillance to manage that risk, either within the Program or through referral to appropriate services.




SUMMARY OF EVALUATION METHODOLOGY

The Australian Health Ministers’ Advisory Council (AHMAC) agreed to an Evaluation of the BreastScreen Australia Program (the Evaluation) in October 2005. The Evaluation addresses the following objectives endorsed by AHMAC in June 2006:
    - assess the outcomes delivered by the BreastScreen Australia Program (the Program);

    - assess the extent to which the Program has achieved its aims and objectives;

    - assess the appropriateness, efficiency and effectiveness of the Program;

    - assess and address the ongoing and unresolved issues impacting on the Program; and

    - identify opportunities to improve the Program overall.


To achieve these objectives, the Evaluation focused on three outcomes:

    - health outcomes: the benefits and risks of the Program;

    - process outcomes: the efficiency of implementation of the Program; and

    - economic outcomes: the cost-utility, cost-benefit and cost-effectiveness of the Program.


AHMAC established an expert Evaluation Advisory Committee (EAC), comprising Australian and international experts and jurisdictional and consumer representatives, to direct and advise on the Evaluation. The EAC developed an Evaluation plan, which was endorsed by AHMAC in March 2007.

Ten projects analysing different aspects of the Program were commissioned to assess Program outcomes. EAC members acted as sponsors to projects, providing expert advice to consultants and reviewing project findings. The Evaluation was also informed by consultation and communication with stakeholders. This report draws together and synthesises findings from the projects and stakeholder communication, and provides recommendations for consideration by AHMAC. Details of each of the projects are provided in Appendix 16.2.


SUMMARY OF EVALUATION FINDINGS AND CONCLUSIONS

Key findings

    1. BreastScreen Australia is broadly available, accessible and acceptable to many women.

    2. The Program has been successful in reducing mortality from breast cancer at the current participation rate of 56% in the target age group (women aged 50–69 years) by approximately 21–28%.

    3. The Program is cost-effective at an approximate cost of $38,000 per life year gained (LYG).3

    4. The Program participation target of 70% of women in the target age group has not been met. Program participation for women aged 50–69 years has been steady at around 56% over the last few years. However, the absolute number of women in the target age group screened increased by 41% between 1996–1997 and 2004–2005. Rescreening rates are low, with only 60.5% of women returning to the Program within the recommended 2-year period following their first screen.

    5. Evidence indicates that mammographic screening of women aged 50–69 years reduces mortality from breast cancer, and supports the continued biennial screening of women at population risk aged 50–69 years. There is limited new evidence from the literature with which to determine the appropriate policy for women aged 40–49 years and over 70 years of age.

    6. Program performance data and results of the Mortality Ecological Study indicate that the greatest mortality benefit is achieved within the target age range of 50–69 years and that the greatest potential improvement in mortality benefit would be achieved with increased participation of women aged 50–69 years.

    7. Participation in the Program reduces treatment-related morbidity. Breast cancers detected through BreastScreen Australia are significantly more likely to be smaller than those diagnosed outside the Program, and a higher proportion of breast cancers detected by BreastScreen Australia are treated by breast conserving surgery (This finding derives from data obtained from the Victorian Cancer Registry 2005-2006. Data from the Royal Australasian College of Surgeons (RACS) (NBOCC 2009) are also supportive of this finding).

    8. For women aged 40–44 years, there is limited evidence of benefit for mammographic screening in relation to mortality reduction. There is also evidence of harms associated with screening in this age group, with a higher rate of invasive investigation without cancer present compared to women aged 45 years and over.

    9. For women aged 45–49 years and women aged 70–74 years, there is some evidence of the benefit in relation to reduction in mortality from breast cancer assoiated with screening.

    10. For women aged 75 years and over there is limited evidence of benefit in relation to mortality reduction associated with screening.

    11. Where there is clear national policy direction, jurisdictional screening policies are consistent with and align with national policy. Where there is a lack of national policy direction, as in the screening of women at elevated risk of breast cancer, there is wide variation in screening practice at jurisdictional level, leading to inequity for women.

    12. Claims have been made previously that mammography reimbursed through the Medicare Benefits Schedule (MBS) impacts on Program participation rate. It does not appear, however, that large numbers of women in the target age group are accessing non-diagnostic mammography outside the Program.

    13. There are currently significant capacity issues in the Program, as evidenced by an increase in delays in recalling women to assessment, falling rescreening rates and plateauing of the participation rate for the target group, despite reduced attendance rates of eligible age groups.

    14. Modelling of Program demand and capacity into the future indicates that, without intervention, demand for the Program, largely as a result of growth in the target cohort, will continue to exceed capacity. Although implementation of digital mammography will improve capacity, its introduction alone will not provide sufficient capacity to address the growing gap between capacity and demand. Radiography workforce issues are the greatest constraint to capacity.

    15. Current governance and management arrangements at the national level have some strengths, notably clear and specific goals and objectives for the Program and robust structures for accreditation. However, there are significant areas requiring improvement including better collaborative decision making and strategic planning at a national level, evidence-based policy development and implementation, and timely response to address new and emerging issues.

    16. The BreastScreen Australia accreditation system is highly accepted and respected. Some aspects of the accreditation system, however, are a burden for services. Program quality assurance would benefit from a greater focus on continuous quality improvement activities, including improved monitoring, in addition to accreditation.

    17. The Program has a wealth of data that are currently underutilised. Despite significant efforts to standardise data collection, inconsistencies remain, limiting usability of data at a national level. There has been limited research regarding the use of Program data to inform policy.


MORTALITY, MORBIDITY AND PERFORMANCE

The BreastScreen Australia Program has met its key objective of reducing morbidity and mortality from breast cancer among the target group of women aged 50–69 years. The Program is broadly available and well accepted by a majority of women with an average national participation rate of 56%. While positive health outcomes have been delivered for women, significant challenges exist that limit Program effectiveness. These include capacity constraints, which are apparent now and will deteriorate over the coming decade, impacting on participation rates, as well as deficiencies in current governance and management arrangements. The Program must address these challenges to remain successful, and needs to focus resources on screening policies aimed at achieving the best mortality and morbidity outcomes for women.

Biennial screening using mammography through the Program at the current participation rate of 56% is associated with a significant reduction in breast cancer mortality at the population level for women aged 50–69 years in the order of 21–28%. The Program is cost-effective at an approximate cost of $38,000 per life year gained (LYG). The breast cancer detection rate is consistent with the national accreditation standard (NAS) and has improved significantly since the Program began, indicating improved performance in cancer detection and assessment of screen-detected abnormalities. More than half of the cancers detected by the Program are small (≤15mm), leading to less aggressive treatment and resulting in improved survival rates.

Screening has both benefits and harms, and all population screening programs aim to minimise harms and maximise benefits of participation. For women in the target age group, the Program is performing well in minimising false-positive and false-negative results, and the recall to assessment rate meets national standards. Sensitivity of mammography is significantly higher in women 70 years and over in the Program. For women aged 40–49 years, Program sensitivity is lower and the rate of false negatives is higher compared with that for older women. The rate of invasive investigation to identify cancers is higher for women aged 40–44 years compared with those aged 45–49 years.

PARTICIPATION

BreastScreen Australia is well accepted by women. National participation has increased for women in the target age group from 51.4% in 1996–1997 to 56.2% in 2004–2005. Participation is similar across socio-economic groups and rural, regional and metropolitan areas. However, for women living in very remote areas, Aboriginal and Torres Strait Islander women and women who speak a language other than English at home, participation rates are significantly lower than the national rate, despite improvements over time.

A high participation rate in the target age group is necessary to achieve substantial reductions in breast cancer mortality for Australian women and to ensure efficient use of resources. The Program objective of a 70% participation target for women in the target age group has not been met. Rescreening rates are also low, with only 60.5% of women returning to the Program within the recommended 2-year period following their first screen.

Failure to achieve the participation target is related to a range of factors, including women choosing not to attend or return, lack of awareness of the Program, as well as a decline in Program capacity. Other factors that may discourage participation include poor understanding among some women about the benefits of mammography screening and low awareness of the increasing risk of breast cancer with increasing age. A positive first-time screening experience, where pain, embarrassment and discomfort are well-managed, strongly influences a woman’s decision to return for a subsequent screen. Evidence suggests that mammography funded through the Medicare Benefits Schedule (MBS) has an insignificant impact on participation rates.

While the 70% participation target provides a level for the Program to strive towards, it should not be considered the ultimate or ‘gold’ standard. A higher participation rate for women in the target age range would result in even greater mortality reduction and cost-effectiveness.

CAPACITY

Demand for the BreastScreen Australia Program has increased over time, with 41% more women in the target group screened in the 2004–2005 screening period compared with 1996–1997. However, Program capacity is reaching its limit, with standards for timeliness of assessment, timeliness of screening appointments and rescreening rates not being met. Time to assessment is of particular concern, with 20% of women recalled for assessment not assessed within 28 days of a screen-detected abnormality.

Modelling to 2027 indicates that population growth will result in an ongoing increase in demand for services over the next 20 years. Digital mammography will provide additional capacity but not enough to fill the growing gap. Capacity has been channelled to women in the target age group of 50–69 years through protocols that give preference to this group. The proportion of women in the target age group participating in the Program has increased over time. However, close to one-quarter of Program capacity is taken up by women in the eligible age group (40–49 years and 70 years and over).

Workforce issues are the greatest constraint to capacity and there is wide variation in productivity across the Program. Small incremental improvements in the number of women who can be screened each year per radiographer can yield significant capacity increases, which could be used to minimise capacity gaps.

GOVERNANCE AND MANAGEMENT

Governance and management arrangements to date have supported the BreastScreen Australia Program with clear aims and objectives and an effective accreditation system. However, these arrangements have not provided strong enough leadership and direction. Program leadership must change to ensure the outcomes of this Evaluation inform future development. The focus must be on timely, collaborative decision-making in response to emerging issues, improved strategic planning and more effective use of Program data to inform Program policy.

The Program accreditation system, while effective, is perceived as onerous and not fully transparent. A review of the accreditation system is required to minimise the burden on services and improve openness of decision making.

The Program has a wealth of data that are currently underutilised. While efforts have been made to standardise data collection, access to data and data analysis have been limited by data inconsistencies, which result in difficulties with national data aggregation. The Program would benefit from the availability of a national data set and a planned data analysis and research program to inform Program policy and future development effectively.

POLICY

There is high-quality evidence that screening women in the target age group of 50–69 years reduces mortality from breast cancer. Meta-analyses of randomised controlled trials (RCTs) undertaken in the 1970s and 1980s found that screening women aged 50–69 years resulted in a reduction in mortality from breast cancer in the order of 25% among those invited to screening and 35% among screening participants.

Limited new evidence is available to inform recommendations for screening younger and older women. Evidence from RCTs is consistent with a 15% reduction in mortality from breast cancer for annual screening of women aged 40–49 years. The risk reduction is less than that for women aged 50–69 years and has not reached statistical significance. Screening performance outcomes are poorer for women aged 40–44 years, meaning that the balance of benefit-to-harm for women in this age group is not as favourable as for women aged 45 years and over. No studies allow assessment of the effectiveness of screening in women 40–44 years compared with those aged 45–49 years. However, the mortality reduction associated with screening increases with age and it could be assumed that women aged 45–49 years would have a mortality benefit closer to that of women aged 50–55 years than women aged 40–44 years. Results from the Mortality Ecological Study undertaken as part of the Evaluation using BreastScreen Australia data support this assumption, although the results should be interpreted with some caution due to the small numbers in the study.

Evidence for the effectiveness of screening is also limited for women aged 70–74 years. Relative risk reduction in mortality from breast cancer has been demonstrated in women aged 65–74 years. Results from the Mortality Ecological Study support evidence of an effect in the order of a 12% reduction in mortality at a population level with a participation rate of 60%. However, the confidence intervals are wide due to small numbers in the study. Program performance data indicate that sensitivity of screening increases with age. A mortality benefit was not demonstrated for women aged 75 and over, although again the results should be interpreted with some caution due to the small numbers in the study.

Improving the participation rate for women in the current target age group of 50–69 years would produce the most improved outcomes in relation to mortality benefit and cost-effectiveness. There may be benefit in extending the target age range to include women aged 45–49 years and 70–74 years. There is some support for this approach from results of RCTs, evaluation of breast cancer screening services in Australia and internationally, and from Program performance data. These same sources provide support for ceasing access to the Program for women aged 40–44 years and women aged 75 years and over.

There is no evidence of benefit or harm from either extending the current screening interval from 2 years to 3 years or reducing the screening interval to 1 year for women aged 50–74 years. These findings support the continued biennial screening of women aged 50–74 years. For women aged 45–49 years, however, the greater likelihood of missing cancers, shorter sojourn time of breast cancer and subsequent higher interval cancer rate suggest that an annual screening interval may be appropriate for this age group, with the caveat that this will also lead to a higher recall rate and higher rate of false-positive results.

Program data indicate that around 17% of Program participants report a family history, past history of breast cancer or present with symptoms of breast cancer. Jurisdictional policy on approaches to screening or excluding these women varies greatly, with the potential for inequitable outcomes and the exclusion of some women from the Program. In some cases, a policy of annual screening has been used for women without discrimination regarding the level of risk. For some of these women, annual screening may not be necessary and this policy can impact on Program capacity. The Program would benefit from implementation of consistent and evidence-based screening policies for women at elevated risk of breast cancer. Strengthened governance and management arrangements would support the implementation of such policies.

While not examined in this Evaluation, there would be value in exploring other policy and service-delivery models for the Program aimed at improving efficiency and quality of services for women. This could include examining possible roles for the Program in risk assessment and management of women at very high risk of breast cancer, diagnostic assessment and greater co-operation or co-location with clinical breast cancer services.

SUMMARY

BreastScreen Australia is, and will continue to be, a cost-effective means of breast cancer control in Australia into the future. Mammography continues to be the most effective tool for breast cancer screening. No other technology has been demonstrated to reduce the breast cancer mortality of women in the population screening setting.

In this section