Evaluating the performance of National Comprehensive Cancer Network (NCCN) breast and ovarian genetic/familial high risk assessment referral criteria for breast cancer women in an Asian surgical breast clinic

Introduction

Breast cancer is the most common cancer among women worldwide, and genetic disposition accounts for about 5–10% of breast cancer cases (1). As a result, the National Comprehensive Cancer Network (NCCN) breast and ovarian genetic/familial high risk assessment version 1.2016 has proposed a set of criteria for further genetic risk evaluation for high risk breast cancer patients (2). While the NCCN criteria were designed to be broad ranging to avoid missing high risk patients, in reality there is not a one-size-fits-all set of criteria, given the differences in clinical settings and resources. In fact, the non-specific nature of some of the NCCN criteria can result in an inordinate number of unnecessary lower risk patient referrals to the genetics clinic, imposing a substantial burden on centers with limited resources and resulting in extended appointment waiting times for the truly high-risk patients.

In our high-risk genetics clinic, we assess a patient’s need for genetic testing using the Manchester score in conjunction with the Breast and Ovarian Analysis of Disease Incidence and Carrier Estimation Algorithm (BOADICEA) risk assessment system as these scoring systems have been validated widely (3) and had been found to correlate well with our local mutation carrier status (4). Hence, it is reasonable to use these same scoring systems as reference standards to assess the sensitivity and specificity of individual NCCN criteria for identifying high-risk patients warranting referral for genetic evaluation.

Though these scoring systems (Manchester using a cut off of ≥15 and BOADICEA ≥10%) may accurately predict the likelihood of mutation carrier, they are complex and can be time consuming to execute. A single NCCN criterion sufficiently predictive of high risk but with greater specificity for screening out lower risk patients, with correlation to the Manchester and BOADICEA scoring systems, would help the referring clinician in a busy surgical clinic to streamline referrals and potentially reduce the volume of lower risk referrals to a high volume genetics clinic.

Methods

A retrospective medical record review of patient data was derived from a prospectively collected database at the Breast Department, KK Women and Children’s Hospital, Singapore. All patients diagnosed with breast cancer from 1 January, 2009 to 31 December, 2011 at our institution were included in the study. In particular, family history information including incidence of breast, ovarian, and other cancers, age at diagnosis and relationship to the proband was collected. Patients with incomplete data, who were not of Asian race or lost to follow up were not included in the study. Patients were followed up till 31 December 2015.

Study patients were assessed using the NCCN criteria for further genetic risk evaluation, and the Manchester and the BOADICEA Web Application version 3 (BWA v3) scoring systems. Patients with Manchester score ≥15 or BOADICEA risk calculation ≥10% were identified as a high risk subgroup qualifying for genetic testing.

Information obtained from this designated high risk subgroup of patients included the number of patients who underwent genetic testing and the outcome of the genetic testing. We compared each of the NCCN referral criteria against the Manchester and BOADICEA results of our breast cancer patients to determine how each of the NCCN criteria fared.

Based on the Manchester and/or BOADICEA scoring systems, each NCCN criterion was assessed for statistical significance and clinical utility in identifying high risk patients. Logistic regression in conjunction with receiver operating characteristic (ROC) analysis were used to identify those NCCN criteria that were statistically significant predictors of high risk patients and to assess the sensitivity and specificity of individual criteria using Manchester and/or BOADICEA as the reference standard. ROC curves were obtained and area under the curve (AUC) calculated as an overall measure of accuracy. Positive and negative predictive values (PPV, NPV) were calculated. Statistical analysis was performed using SAS V9.3 (SAS Inc., Cary, NC, USA).

This study obtained ethics approval from SingHealth Centralised Institutional Review Board (CIRB Ref 2015/2188).

Results

A total of 666 female patients with breast cancer were identified in the database during the study period. Thirty-one patients were lost to follow up and one patient of Caucasian race was excluded from the study. The majority of study patients were Chinese (80%), followed by Malay (10%), Indian (6%) and other races (4%) (Table 1). Fifty-three percent (53%) of patients were ≥50 years old, and median age at breast cancer diagnosis was 51. Seven patients (1.05%) had bilateral breast cancers, and 63 patients (9.46%) had a family history of breast/ovarian cancer.

Table 1 Demographics of breast cancer patients identified in the database from 2009 to 2011
Full table

Using the NCCN criteria of early age onset ≤50 for further genetic risk evaluation, up to 47% of breast cancer patients would have been referred to the genetics clinic for further assessment. None of our study patients qualified for further genetic risk evaluation under criteria of ovarian cancer, known mutation in a cancer susceptibility gene within the family, male breast cancer, patients genetically predisposed to higher risk such as Ashkenazi Jewish descent, or personal and/or family history of three or more cancers listed in Table 2, criteria 3.

Table 2 Breast cancer patients identified in the database from 2009–2011 using NCCN guidelines
Full table

Using a Manchester cutoff score of ≥15, 12/666 (1.80%) patients were classified as high risk and using BOADICEA ≥10% probability of BRCA1 and BRCA2 mutation, 17/666 (2.55%) were classified as high risk, resulting in 20/666 (3.00%) patients classified as high risk by either Manchester or BOADICEA who qualified for genetic testing. Six hundred forty-six (97.00%) patients were not high risk by either criteria.

Of patients with Manchester Score ≥15 or BOADICEA ≥10%, 9 were high risk by both Manchester and BOADICEA, 8 were positive by BOADICEA only, and 3 were positive by Manchester only.

Mean ± SD age of patients indicated as high risk by Manchester only (M+), BOADICEA only (B+), both (MB+) and neither (N), was 51.7±8.62, 43.0±11.5, 43.2±9.55 and 52.6±11.3 respectively (P=0.070). Percentage of Chinese by M+, B+, MB+ and N was 66.7, 75.0, 88.9 and 80.5 (P=0.332), and respective mean ± SD follow-up times were 4.7±1.15, 4.4±0.74, 4.4±1.13 and 4.2±0.59 years (P=0.749).

Of the patients scoring positive by Manchester or BOADICEA, only one underwent genetic testing and was found to have the BRCA 1 c.67-68 delins AG mutation. The patient had been on Tamoxifen since surgery and was well with no evidence of recurrence or metastasis. The other 19 positive patients did not undergo genetic testing, and all were found to be well except one whose cancer had metastasized. This patient had a triple negative tumor and was positive by Manchester and BOADICEA for BRCA 1.

Using univariate logistic regression, three NCCN criteria were identified as statistically significant individual predictors of patients classified as high risk by either Manchester or BOADICEA scorings. These were (I) early age onset ≤50 breast cancer (II) breast cancer at any age and 1 or more close blood relatives with breast cancer at age ≤50, and (III) breast cancer at any age and 2 or more close blood relatives with breast cancer and/or pancreatic cancer at any age (Table 2). Of the three NCCN criteria, breast cancer at any age and 1 or more close blood relatives with breast cancer at or younger than 50 years old was statistically significant as a predictor of high risk (P<0.0001) with sensitivity =70%, specificity =96% and AUC =0.83 (Figure 1). Percentage of high risk patients, based on the two scoring systems, among our cohort was 3% resulting in PPV =37% and NPV =99%. Statistical performance for the other two criteria was as follows: (I) early age onset breast cancer ≤50: sensitivity =0.75, specificity =0.56 and AUC (95% CI) =0.66 (0.56, 0.76); (II) breast cancer at any age and 2 or more close blood relatives with breast cancer and/or pancreatic cancer at any age: sensitivity =0.40, specificity =0.99 and AUC (95% CI) =0.70 (0.59, 0.81) (Table 3).

Figure 1 ROC curve of NCCN criteria: breast cancer at any age and ≥1 close relative with breast cancer ≤50. ROC, receiver operating characteristic; NCCN, National Comprehensive Cancer Network.

Table 3 NCCN criteria as univariate predictors of referral to genetics clinic
Full table

Discussion

NCCN breast and ovarian genetic/familial high risk assessment has proposed referral criteria for further genetic risk evaluation. However, it has multiple criteria and the generality of some of the criteria can result in over-diagnosis for referral to the high risk genetics clinic. The nature and consequences of the inherent generality is exemplified by criterion 1b (Table 2), that is, breast cancer early age onset ≤50, which resulted in a 47% for referral to the high risk genetics clinic. Of course, generality in a criterion translates to high sensitivity and high negative predictive value, with the intention that exceedingly few high-risk patients will fail to be identified. However, in countries outside of Europe and United States (and even in these countries), cancer genetics clinics are often under-resourced, such large numbers of patients meeting referral criteria may become problematic due to limitations in workload capacity and preclude other patients with higher likelihood of testing positive from receiving timely care.

In our study population, using the Manchester and/or BOADICEA scoring (Manchester ≥15 and/or BOADICEA ≥10%) as the reference standard for high risk, the best single NCCN criterion which correlated well with the scoring systems for identifying high risk patients, was breast cancer at any age and ≥1 close blood relatives with breast cancer at or younger than 50 years old (sensitivity 70.0%, specificity 96.3%), AUC 0.83 which was the highest among the three statistically significant criteria. Though early age onset ≤50 years old had the highest sensitivity among the three criteria, it also has the lowest specificity. This could potentially lead to large number (up to 47% in our study) of patients being referred to the genetics clinic with a low likelihood ratio for a positive test result. The other criteria breast cancer at any age and ≥2 close blood relatives with breast cancer and/or pancreatic cancer at any age, on the other hand, was most specific but only had a sensitivity of 40%, hence compromising on its use as a screening criteria.

By using the criterion of breast cancer at any age and >1 close blood relatives with breast cancer at or younger than 50 years old as the screening question in our surgical clinic, we could potentially reduce both screening time (a single criterion) and the number of unnecessary referrals, while achieving an acceptably low false negative rate (1—NPV), i.e., failing to identify patients that are truly high risk. This is the first study, to our knowledge, which seeks to identify and evaluate specific components of the NCCN guidelines for further genetic risk evaluation in an Asian breast cancer patient population using the Manchester and/or BOADICEA scoring systems as a reference standard.

BOADICEA was used in our setting because it has been known to be the most reliable scoring tool in predicting mutation carrier probabilities compared to other scoring systems (3,5). It also has the added advantage of including the tumor pathologic subtype data such as triple negative subtypes, to enhance its predictive screening capability (6). However, it is not without limitations. It was not designed for use in patients with ductal carcinoma in situ (DCIS) (7) which we had 9.60% in our study population. It is also not accurate in the extreme age ranges of <20 or >80 years old (8) which we fortunately only had none and 1% in our study population respectively. In addition, it requires computation which is time and labor intensive.

The Manchester Scoring system, on the other hand, is an assessment tool that does not require computation as does BOADICEA. It is simple and has been widely validated in the United Kingdom (9,10). Though easier to use, it still relies heavily on the age of diagnosis of cancer and the documentation of specific cancers in the family to get an accurate score, making it time consuming for the busy clinician to get a comprehensive family history in a surgical setting.

Another limitation of the Manchester and BOADICEA scoring systems is its tendency to underestimate the number of mutation carriers in families with limited family structure (11) which may be a problem in our cohort with <10% of the patients having a documented family history of cancers. In addition, in the Asian population (12), Manchester and BOADICEA scoring systems have been reported to have a lower sensitivity, specificity and PPV as compared to the Caucasian population. Though not the most ideal scoring systems catered for the Asian population, Manchester (cut off ≥15) and BOADICEA have been reported to have somewhat reasonably acceptable sensitivities (72%, 40%) and specificities (74%, 85%) respectively in the Asian population (12). In fact, BOADICEA has been reported to have the highest AUC of 0.8 among other scoring systems in an Asian population though it tends to underestimate the number of BRCA1/2 carriers (13).

Due to the complexity of the Manchester and BOADICEA scoring tests which can be time consuming to perform in the busy surgical clinic, it may be best that those deemed high risk patients by either of these scoring tests are referred to the genetics clinic for further evaluation. It has also been shown that patients referred to the high risk clinic for genetic counseling prior to the genetic testing tend to have greater satisfaction and understanding (14). It then becomes paramount that referral to the high risk clinic be guided by appropriate criteria. NCCN criteria for further evaluation of genetic risk are comprehensive but have shortcomings as some criteria are too general. Therefore, we wanted to identify which of the NCCN criteria were most indicative of high risk in our local breast cancer patients where the majority does not have a family history of breast cancer, by evaluating the NCCN criteria with Manchester and/or BOADICEA.

In an attempt to identify a wider scope of high risk patients who will require genetic testing, we used the two scoring systems, instead of choosing on one system or the other. Based on a positive result from either or both scoring systems, we identified three additional high risk patients compared to using either system alone. We specifically chose the Manchester and/or BOADICEA scoring systems in our study because these two systems have been previously shown to correlate well with the actual mutation status in our local population (4).

In our study, only 1 of the 20 patients identified as high risk presented for genetic testing. Declining genetic testing is a common problem in our clinical setting. There are many reasons why genetic testing is declined, with up to 42% refusing genetic testing in one study (15). Cost of genetic testing along with possible perceived insurance discrimination (15) and inability to cope with an undesirable test result (16) were often quoted as some of the reasons. In a study by Godard et al. (17), it was estimated that up to 36.3% of the patients who declined genetic testing quoted fear of psychological effects of genetic testing as the most common reason. In our case, the poor uptake rate for genetic testing was because the study predated the restructuring of our cancer genetics services in 2014 which increased the workload capacity tremendously (18). Though ideally all the patients should get genetic testing as the gold standard to assess the applicability of each NCCN criteria, this was not possible in this retrospective study as not many patients had genetic testing in that era. We instead used the next best available option of applying the Manchester and/or BOADICEA scoring to model what was the likelihood of actually being positive, which has been shown to be reliable in a recent prospective study by our centre (4).

Based on the Manchester and/or BOADICEA scoring system, our calculated proportion of high risk patients was 3.0%. Assuming that these high risk patients were tested positive, this figure is still considerably lower than that reported in the literature. The reported prevalence of BRCA1 and BRCA2 were 10.9% and 7.6%, respectively (3), in the United Kingdom. In a local study by Ang et al. (19), the prevalence of BRCA1 and BRCA2 mutations were 6.7% and 8.9%, respectively. However, they used the BRCAPRO model coupled with a genetic counselor’s assessment to identify the high risk patients, instead of the Manchester and/or BOADICEA scoring as in our study. In another more recent local study, the prevalence of BRCA1 and BRCA2 was 10.5% and 7.7%, respectively (4). This is not surprising, and the lower high risk prevalence rate in our study reflects the fact that this study was done in a surgical breast clinic setting, in contrast to those done at high risk clinics which was often the case reported in literature. There have been no studies, to our knowledge, investigating prevalence of high risk patients based on both Manchester and/or BOADICEA scorings in a surgical breast clinic.

Our study is not without limitations. We assumed that a cutoff Manchester score of ≥15 and/or BOADICEA ≥10% will be adequate to identify virtually all the high risk patients suitable for genetic testing with an acceptable false negative rate. We also could not compensate for the fact that the majority of our patients in this retrospective cohort did not have genetic testing. However, this study only aims to evaluate the performance of the Manchester and/or BOADICEA scoring systems with NCCN referral criteria. Finally, we had no patients with male breast cancer, ovarian cancer, known mutation in a cancer susceptibility gene within the family, patients from a population at increased risk such as Ashkenazi Jewish descent and patients with a personal and/or family history of three or more of the following of pancreatic, prostate cancer, sarcoma etc., as specified in the NCCN criteria. Fulfillment of any of these rarer criteria would increase the probability of the patient being referred to the high risk genetics clinic, but we were unable to assess this probability in our study population. As a result, we are unable to extrapolate the applicability of our study results in the above mentioned specific patient subgroups.

This study provides considerable insight on the applicability of the NCCN referral criteria, based on the Manchester and/or BOADICEA scoring systems in our local setting, as a clinically useful and feasible approach for diagnosing high risk patients warranting referral for genetic testing. Our purpose is to reduce the number of low a priori risk referrals while holding the false negative rate appropriately low in patients who have high a priori risk. As we progress towards fine tuning the referral criteria depending on individual setting and available resources, this paper will certainly help guide future designs of more stringent guidelines.

Conclusions

Among the NCCN criteria, the criterion which correlated best with Manchester ≥15 and/or BOADICEA ≥10%—is breast cancer at any age and ≥1 close blood relatives with breast cancer at or younger than 50 years old which has sensitivity =70%, specificity =96.3%, PPV =36.8% and NPV =99.0% in our Asian population setting. This criterion may be useful as a simple yet effective screening tool for the clinician in a busy surgical breast clinic setting to guide appropriate referral to the genetics clinic. We are not suggesting that it can replace the NCCN guidelines; however, in a busy surgical clinic, where there is insufficient time to rehearse the entirety of the NCCN criteria with a patient, a single ‘best’ NCCN criterion—in conjunction with clinical experience, expertise and judgment—will substantially reduce the work burden from a priori low risk referrals to the genetics clinic while maintaining the false negative rate at an acceptably low value (1.0%). We believe that as the cost and awareness of genetic testing evolve, so must the referral criteria guidelines evolve to best fit the clinical setting. As awareness improves, the identification of patients at risk will increasingly be borne by general surgical practices and family doctors. This study will certainly be an informative guide to clinicians and policy makers alike as the practice evolves.

Acknowledgements

We would like to acknowledge Mrs. Met-Domestici Marie, our genetic counselor who previously worked at National Cancer Centre (Singapore) for teaching Dr. Borje on the use of the Manchester and BOADICEA scoring systems so that an accurate score for the patients could be calculated. Thanks go also to Miss Shao-Tzu Li who verified the number of patients who underwent genetic testing and her helpful inputs.

Footnote

Conflicts of Interest: The authors have no conflicts of interest to declare.

Ethical Statement: This study obtained ethics approval from SingHealth Centralised Institutional Review Board (CIRB Ref 2015/2188).

References

1. Febbraro T, Robison K, Wilbur JS, et al. Adherence patterns to National Comprehensive Cancer Network (NCCN) guidelines for referral to cancer genetic professionals. Gynecol Oncol 2015;138:109-14. [Crossref] [PubMed]
2. NCCN Clinical Practice Guidelines in Oncology (NCCN Guidelines) Genetic/Familial High-Risk Assessment: Breast and Ovarian. National Comprehensive Cancer Network; version 1.2016. Available online: http://www.nccn.org/professionals/physician_gls/pdf/genetics_screening.pdf, accessed January 15, 2016.
3. Antoniou AC, Hardy R, Walker L, et al. Predicting the likelihood of carrying a BRCA1 or BRCA2 mutation: validation of BOADICEA, BRCAPRO, IBIS, Myriad and the Manchester scoring system using data from UK genetics clinics. J Med Genet 2008;45:425-31. [Crossref] [PubMed]
4. Wong ES, Shekar S, Met-Domestici M, et al. Inherited breast cancer predisposition in Asians: multigene panel testing outcomes from Singapore. npj Genomic Medicine 2016;1:15003.
5. Fischer C, Kuchenbäcker K, Engel C, et al. Evaluating the performance of the breast cancer genetic risk models BOADICEA, IBIS, BRCAPRO and Claus for predicting BRCA1/2 mutation carrier probabilities: a study based on 7352 families from the German Hereditary Breast and Ovarian Cancer Consortium. J Med Genet 2013;50:360-7. [Crossref] [PubMed]
6. Mavaddat N, Rebbeck TR, Lakhani SR, et al. Incorporating tumour pathology information into breast cancer risk prediction algorithms. Breast Cancer Res 2010;12:R28. [Crossref] [PubMed]
7. Ståhlbom AK, Johansson H, Liljegren A, et al. Evaluation of the BOADICEA risk assessment model in women with a family history of breast cancer. Fam Cancer 2012;11:33-40. [Crossref] [PubMed]
8. Antoniou AC, Cunningham AP, Peto J, et al. The BOADICEA model of genetic susceptibility to breast and ovarian cancers: updates and extensions. Br J Cancer 2008;98:1457-66. [Crossref] [PubMed]
9. Evans DG, Eccles DM, Rahman N, et al. A new scoring system for the chances of identifying a BRCA1/2 mutation outperforms existing models including BRCAPRO. J Med Genet 2004;41:474-80. [Crossref] [PubMed]
10. Evans DG, Lalloo F, Wallace A, et al. Update on the Manchester Scoring System for BRCA1 and BRCA2 testing. J Med Genet 2005;42:e39. [Crossref] [PubMed]
11. Barcenas CH, Hosain GM, Arun B, et al. Assessing BRCA carrier probabilities in extended families. J Clin Oncol 2006;24:354-60. [Crossref] [PubMed]
12. Thirthagiri E, Lee SY, Kang P, et al. Evaluation of BRCA1 and BRCA2 mutations and risk-prediction models in a typical Asian country (Malaysia) with a relatively low incidence of breast cancer. Breast Cancer Res 2008;10:R59. [Crossref] [PubMed]
13. Kwong A, Wong CH, Suen DT, et al. Accuracy of BRCA1/2 mutation prediction models for different ethnicities and genders: experience in a southern Chinese cohort. World J Surg 2012;36:702-13. [Crossref] [PubMed]
14. Armstrong J, Toscano M, Kotchko N, et al. Utilization and Outcomes of BRCA Genetic Testing and Counseling in a National Commercially Insured Population: The ABOUT Study. JAMA Oncol 2015;1:1251-60. [Crossref] [PubMed]
15. Peterson EA, Milliron KJ, Lewis KE, et al. Health insurance and discrimination concerns and BRCA1/2 testing in a clinic population. Cancer Epidemiol Biomarkers Prev 2002;11:79-87. [PubMed]
16. Bleiker E, Wigbout G, van Rens A, et al. Withdrawal from genetic counselling for cancer. Hered Cancer Clin Pract 2005;3:19-27. [Crossref] [PubMed]
17. Godard B, Pratte A, Dumont M, et al. Factors associated with an individual's decision to withdraw from genetic testing for breast and ovarian cancer susceptibility: implications for counseling. Genet Test 2007;11:45-54. [Crossref] [PubMed]
18. Tan RY, Met-Domestici M, Zhou K, et al. Using Quality Improvement Methods and Time-Driven Activity-Based Costing to Improve Value-Based Cancer Care Delivery at a Cancer Genetics Clinic. J Oncol Pract 2016;12:e320-31. [Crossref] [PubMed]
19. Ang P, Lim IH, Lee TC, et al. BRCA1 and BRCA2 mutations in an Asian clinic-based population detected using a comprehensive strategy. Cancer Epidemiol Biomarkers Prev 2007;16:2276-84. [Crossref] [PubMed]