Development and validation of an ultrasound-based nomogram for preoperative prediction of cervical central lymph node metastasis in papillary thyroid carcinoma

Introduction

Thyroid cancer is the fifth most commonly diagnosed cancer among women and, in the United States, accounts for 11% of malignancies in adolescents (1). Papillary thyroid carcinoma (PTC) accounts for approximately 80% of all thyroid carcinoma cases. Despite having an indolent clinical course, PTC is associated with cervical lymph node (LN) metastasis. It is estimated that up to 90% of patients with PTC develops central lymph node metastasis (CLNM), which occurs at the early stage (2). CLNM, which is associated with distant metastasis, recurrence, and reduced survival (2-4), is usually treated through central compartment neck dissection (CCND). Some studies have shown that CLNM is present in 45% of PTC patients with clinically negative central compartment LNs (cN0) who undergo prophylactic CCND (5). However, the risk of nerve injury and hypoparathyroidism is heightened with CCND, and much controversy surrounds whether this should be performed prophylactically in cN0 PTC patients (6-9). Therefore, preoperative knowledge of CLNM holds value in determining the suitability of a patient for surgical resection and the need for adjuvant treatment, thereby contributing to better therapeutic strategies.

Ultrasound (US) is the most effective method for preoperatively determining the tumor-node-metastasis stage of PTC, and US features are associated with the aggressiveness and outcome of PTC per tumor-node-metastasis stage (7). Although US has good diagnostic accuracy for cervical lateral LN metastasis (LLNM), due to interference of the thyroid and other adjacent organs, the sensitivity of US in the diagnosis of CLNM is only about 30.0–53.2%, which is far lower than its sensitivity in diagnosing LLNM (70–93.9%) (10-13). These results indicate that a large number of patients who are identified to be cN0 preoperatively may have CLNM (14). Therefore, the independent predictors for CLNM must be identified, and a precise and convenient prediction model by which to stratify patients with PTC preoperatively is built; this will facilitate preoperative staging and risk stratification, including upstaging of cN0 to the appropriate level and assessing the extent of the benefit to the patient.

Furthermore, there has been an increasing amount of attention focused on research into the preoperative diagnosis and prognostic stratification of patients with PTC based on BRAF mutations. BRAF mutations have been widely observed in patients with PTC. The BRAF V600E mutation is a strong risk factor for tumor invasion, LN metastasis, and recurrence (15-18). Many studies have reported an association between the BRAF V600E mutation and the US features, patients’ clinical characteristics, and CLNM (19,20). Some studies have shown the BRAF V600E mutation to be an independent predictor of CLNM in PTC (20). However, the results of the studies to date have been inconsistent (21-23), and further investigation is required.

Although earlier studies evaluated the relationship between US features and CLNM in PTC, they did not validate the predictive value of a prediction model comprising US features in a validation cohort, nor did they produce a model to serve as a practical, visual, operable, and innovative tool to guide clinicians (24-27). Nomograms are appreciable, intuitive, and visual tools that have received increasing recognition and have improved the performance of medical practitioners (22,28-30). Some studies have used nomograms with clinical and postoperative pathological characteristics to predict CLNM; however, in these studies, the risk factors were acquired postoperatively (22,29). Therefore, this study aimed to develop and validate an accessible nomogram that integrates US characteristics with clinical features to predict CLNM in PTC patients preoperatively and allow tailoring of CCND to the individual patient. We present the following article in accordance with the STROBE reporting checklist (available at http://dx.doi.org/10.21037/gs-20-75).

Methods

Patient selection

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). This single-center study was approved by the Guangdong Provincial People’s Hospital Research Ethics Committee (No. GDREC2018315H) and individual consent for this retrospective analysis was waived. The study included a training dataset and a validation dataset. The medical records of the institutional database were searched to identify patients with PTC who underwent thyroidectomy with CCND and whose PTC had been confirmed by histopathology and tested for BRAF V600E mutation status between July 2014 and June 2019. A total of 512 consecutive patients (147 male, 365 female; mean age, 42.74±10.02 years) identified between July 2014 and December 2017 were enrolled as the training dataset. An independent validation cohort comprising 306 patients (83 male, 223 female, mean age 42.71±10.13 years) identified between January 2018 and June 2019 was also enrolled.

The inclusion criteria were as follows: (I) a preoperative US examination performed within one month of the operation; (II) the patient had undergone CCND; (III) CLNM status, as confirmed by postoperative pathology; (IV) a postoperative BRAF V600E mutation test result; (V) no history of thyroid cancer surgery or CCND; and (VI) complete medical data.

US technique and imaging analysis

All US examinations were performed during the study period using diagnostic US instruments (Hitachi, Ltd., Tokyo, Japan), including Avius, Preirus, and Ascendus scanners with real-time high-frequency linear array transducers (6–13 MHz). The preoperative US examination was performed on patients who had thyroid nodules, and the data were stored for later analysis.

The US images were retrospectively and independently analyzed by two experienced radiologists (with 10 and 15 years of experience in thyroid imaging, respectively). The radiologists were blinded to the patients’ information, medical data, pathology, and BRAF V600E mutation test results. When disagreement arose, the final decision was made by a third radiologist (with 20 years of experience in thyroid imaging).

The US features were defined as follows: (I) tumor size (maximum length of the largest carcinoma on the sonogram); (II) multiple (≥2) tumors on US (in this case, the largest tumor was observed); (III) solid or predominantly solid tumor (solid components occupying no less than 50% of the tumor) (31); (IV) very hypoechoic/hypoechoic (reduced echogenicity compared with the neighboring neck muscle or thyroid parenchyma) (31); (V) a poorly defined margin (challenging to distinguish the tumor edge from the thyroid tissue) (31); (VI) an irregular shape (a jagged, spiculated, or lobulated border) (31); (VII) punctate echogenic foci in the solid components (<1 mm in diameter) (32); (VIII) capsule contact or involvement (more than 25% of the tumor touching or adjacent to the thyroid capsule or an absence of echogenicity of the surrounding thyroid capsule on US); and (IX) If any of the following features were detected by US, the US-based CLNM/LLNM status was defined as positive: calcification, cystic degeneration, cortical hyperechogenicity, and/or abundant or irregular blood flow within the LN. If there was a disagreement, the LN status was decided by the third radiologist.

BRAF V600E mutation analysis

The FFPE DNA Kit (Amoy Diagnostics, Co., Ltd., Xiamen, China) was used according to the manufacturer’s specifications. A SMA4000 ultraviolet spectrophotometer (Merinton, Inc., Beijing, China) was used to gauge the absorbance of the extracted DNA, and each of the OD260/OD280 values was between 1.8 and 2.0. A human BRAF V600E Mutation Detection Kit (Amoy Diagnostics, Co., Ltd., Xiamen, China) was used to determine the BRAF mutation status. The BRAF mutant gene was augmented using a LightCycler 480 Instrument (Roche Diagnostics, Vienna, Austria) according to the manufacturer’s specifications. The FAM signal was captured, and the BRAF mutation status was assessed by applying the FAM Ct value. The BRAF V600E mutation status was categorized as negative when the FAM Ct value of the sample was >27 or lower than the minimum detectable level of the kit. The BRAF V600E mutation status was categorized as positive when the FAM Ct value of the sample was <28.

Statistical analysis

The patients in the training and validation cohorts were divided into a CLNM-positive group and a CLNM-negative group according to the status of the cervical central LNs reported by pathology. All tests were two-sided, and the criterion for statistical significance was set at 0.05.

The least absolute shrinkage and selection operator (LASSO) method was employed to determine the most useful predictive US characteristics. The US-based nomogram to predict the individual probability of CLNM was then built and plotted based on multivariate logistic regression analysis in the training dataset. Harrell concordance indexes (C-index, which is equivalent to the area under the receiver-operating characteristic curve) were calculated to assess the discrimination performance of the US-based nomogram (model 1) and model 1 with BRAF V600E mutation status (model 2) in the training and validation datasets. Moreover, visual calibration curves were plotted to evaluate the calibration of the models. Decision curve analysis (DCA) was implemented, and the results were plotted to assess the models’ feasibility for use in clinical practice by quantifying the net benefit at various threshold possibilities in the validation cohort (33). The statistical analyses were performed using SPSS 25.0 software (IBM Corp., Armonk, NY, USA) and R software with RStudio version 1.1.463 (RStudio, Inc., Boston, MA, USA).

Results

Clinical and US characteristics

The clinical features and US characteristics of the patients in the training and validation datasets are summarized in Table 1. According to the postoperative histopathological examination, the rates of CLNM positivity in the training and validation datasets were 50.2% and 49.0%, respectively. Therefore, there was no significant difference between these two datasets in terms of the prevalence of CLNM (P=0.745). Furthermore, there was no significant difference in sex or age between the datasets or between the CLNM-positive and CLNM-negative groups, which confirmed the appropriateness of the training and validation datasets used.

Table 1 Clinical and US features of patients in the development and validation datasets
Full table

Selection of US features

Tumor size and shape, punctate echogenic foci, multiple tumors, capsule contact or involvement, US-based LLNM status, and US-based CLNM status were among the US characteristics that showed significant between-group differences in the training dataset (Table 1). Twelve characteristics were reduced to 3 potential risk factors in the 512 patients in the training dataset and were characterized by nonzero coefficients in the LASSO logistic regression model (Figure 1). These three risk factors were tumor size, multiple tumors, and US-based CLNM status.

Figure 1 Selection of ultrasound features using the least absolute shrinkage and selection operator (LASSO) logistic regression model in the training dataset. (A) The 12 ultrasound features are profiled by the LASSO coefficient. A coefficient profile plot against the log (λ) sequence was produced; (B) the area under the receiver-operating characteristic curve was plotted versus log (λ). The minimum criteria, together with 1 standard error of the minimum criteria (1 – SE criteria), was used to draw the dotted vertical lines at the optimal values. A λ value of 0.096, with log (λ) –3.385 was chosen (1 – SE criteria) according to 10-fold cross-validation, where optimal λ resulted in 3 nonzero coefficients.

Performance of the US-based nomogram

Logistic regression analysis revealed sex, age, tumor size, multiple tumors, and US-based CLNM status to be independent risk factors for CLNM (Table 2). All these independent risk factors were incorporated into prediction model 1 to create the US-based nomogram (Figure 2).

Table 2 Risk factors for CLNM in PTC
Full table

Figure 2 Ultrasound-based nomogram predicting the incidence of CLNM. The patients’ values were found on both axes for prediction of the incidence of CLNM with the nomogram. A line was drawn to the point axis to identify the number of points contributing to the variable values. The points for all variables were summed up. The incidence of CLNM could be predicted by finding the sum on the total point line. CLNM, central lymph node metastasis

In the training dataset, the observations and predictions showed high concordance according to the calibration curve of the US-based nomogram for predicting CLNM (Figure 3A). The Hosmer-Lemeshow test revealed a nonsignificant difference (P=0.425), which showed no deviation from an ideal fit. The C-index for model 1 was 0.765 [95% confidence interval (CI): 0.724–0.806] in the training dataset and improved to 0.791 (95% CI: 0.740–0.842) in the validation dataset. In the validation dataset, the incidence of CLNM was verified to have high calibration (Figure 3B). The Hosmer-Lemeshow test revealed no significant differences (P=0.266).

Figure 3 Calibration curves. (A) and (B) show the calibration curves for the model 1 in the training and validation cohorts, respectively. (C) and (D) show the calibration curves for model 2 in the training and validation cohorts. Consistent with the observed results and predicted risks of CLNM, each model’s calibration is portrayed by a calibration curve. Values on the x and y axes show the predictive potential and actual incidence of CLNM, respectively. Perfect prediction based on an idealized model is represented by the diagonal dotted line. The ideal performance of the nomogram is represented by the solid line, and the closer the solid line fits the dotted line, the better the prediction will be. CLNM, central lymph node metastasis

Predictive value of adding BRAF V600E mutation status to the US-based nomogram

The performance of the two models is shown in Table 2. In the training cohort, the C-index for model 2 (0.764) was slightly lower than that of model 1 (0.765). However, in the validation cohort, the C-index for model 2 was slightly higher (0.792). In both cohorts, the observations and predictions showed high concordance according to the calibration curve for predicting CLNM (Figure 3C,D).

Clinical use

The DCA results for US-based CLNM status, and the two models are presented in Figure 4. Based on the DCA results, the net benefits of the two models were comparable, and there was a significant overlap. The decision curve indicates that model 1 can predict CLNM when the threshold probability is between 8% and 82%, which shows that it would be more effective than a treat-none or treat-all strategy.

Figure 4 Decision curve. Values on the y-axis represent the net benefit. The US-based CLNM status and models 1 and 2 are represented by the yellow, blue, and red lines, respectively. The assumption that no patients have CLNM (the treat-none scheme) is represented by the black line and that all patients have CLNM (the treat-all scheme) is represented by the green line. Based on the decision curve, the US-based nomogram could be used to forecast CLNM when a doctor’s or patient’s threshold probability is between 8% and 82%, which would have better performance than the US-based CLNM status. CLNM, central lymph node metastasis; US, ultrasound.

Discussion

Our study combined the US features and clinical risk factors of PTC patients to establish an authentic visual prediction nomogram that can be conveniently and accurately used to stratify patients and expedite prediction of CLNM in individual patients with PTC preoperatively. Young age, male, tumor size, multiple tumors, and US-based CLNM status, the data for which were reached preoperatively, were found to be independent predictors of CLNM. Earlier studies showed that being young, male, having a large tumor size, multifocality, lymphovascular invasion, extrathyroidal extension, and capsular invasions were risk factors of CLNM (2,22). Earlier studies have developed nomograms for predicting CLNM; however, these included clinical and postoperative pathological characteristics and not US features (22,29). Even so, the results were similar to those of our study, which suggests that preoperative US features bear a strong correlation with pathological characteristics.

US elastography (USE) and contrast-enhanced ultrasound (CEUS) are emerging as complementary tools for differentiating carcinoma from benign nodules and may provide additional information to support clinical decision-making (34-36). USE could be used to partially assess features of thyroid nodules, mainly with the use of semiquantitative methods (36). However, the histological features of the nodules and inherent technical limitations of USE may lead to misinterpretations and pitfalls (34). CEUS has the advantage of easing analysis of macro- and microvascularization patterns. Nevertheless, nodule size and overlapping CEUS data may affect the results of examination and interpretation. According to the guidelines of the European Federation of Societies for Ultrasound in Medicine and Biology (EFSUMB), CEUS is not currently recommended for clinical use (35). Finally, although USE and CEUS are promising techniques, they are most likely not available in some ultrasound laboratories and were not incorporated into our study.

Our study included 12 US characteristics as underlying predictors of CLNM in PTC. This group was reduced to a group of three independent predictors via LASSO regression, which is superior to the method of selecting predictive factors based on the outcome reported by univariate analysis. Univariate analysis found tumor shape, capsule contact or involvement, punctate echogenic foci, and US-based LLNM status to be associated with CLNM in PTC; however, these were not considered independent predictors in LASSO regression. Moreover, some studies, including ours, showed both young age and male to be independent predictors of CLNM (22,25,32,33). These outcomes suggest that young male patients with PTC are at increased risk of CLNM.

The preoperative US-based CLNM status is a qualitative US feature that is easy to determine. However, this factor was not included in previous studies of combinations of preoperative US features for predicting CLNM, possibly because of the poor diagnostic efficiency of the US in diagnosing CLNM (11,13,25,37). In the present study, the sensitivity of US-based CLNM status was 44.4% in the training dataset, and 44.0% in the validation dataset and the specificity values were 93.3% and 89.7%, respectively. Our findings show that US-based CLNM status is related to the pathology of the central LNs and is an independent predictor of CLNM.

There are reports in the literature to suggest that BRAF V600E mutation status is an independent predictor of CLNM and a significant prognostic marker that can be preoperatively assessed (15,16,18,20). However, these reports are controversial (22,23). BRAF V600E mutation status was not shown to be able to predict CLNM in univariate analysis (P=0.926), and so this variable was excluded from the model. However, the lack of a statistically significant association with CLNM does not necessarily mean that BRAF V600E mutation status is not essential. Subtle differences in datasets or confounding factors can cause significant predictors to be rejected (38). Therefore, we considered BRAF V600E mutation status as a potential predictor in the process of developing the model.

Nevertheless, the BRAF V600E mutation status did not improve the performance of the US-based nomogram. Tumor heterogeneity may also lead to sampling bias in biopsy results and a higher false-negative rate for BRAF V600E mutation status (39,40). Consequently, determining the BRAF V600E mutation status did not significantly improve the performance of the US-based nomogram in predicting CLNM (41), and a simpler predictive nomogram based on US and clinical features might be appropriate.

Our concise US-based nomogram yielded an agreeable discrimination outcome (C-index of 0.765 in the training dataset and 0.791 in the validation dataset). Chen et al. (24) integrated immunohistochemical factors and BRAF V600E mutation status with US characteristics to detect CLNM in patients with PTC and achieved much higher accuracy, with an area under the curve of 0.865 (95% CI: 0.818–0.912) when the multivariate logistic regression equation was derived. However, their study was inherently limited by a lack of independent validation, and there is still a long way to go before receiver-operating characteristic curve analysis can be integrated into clinical practice for personalized prediction of risk (24,27). Therefore, we proposed and validated a more concise US-based nomogram for predicting the probability of CLNM in individual patients by integrating US characteristics and clinical risk predictors, which can be easily assessed preoperatively. This user-friendly scoring system allows physicians and patients to predict the probability of CLNM before surgery, which fits in with the current trend towards personalized care (28). Clinicians can now perform CCND in patients who have PTC with a high probability of CLNM (22).

The fundamental reason for developing such a nomogram is to determine the probability of CLNM in PTC at the level of the individual patient. However, the prediction, calibration, and discrimination performance are still inadequate to yield the desired outcome in clinical practice. Furthermore, there are limitations in terms of miscalibration and extent of discrimination (38,42). Therefore, we assessed the value of our nomogram in improving clinical decision-making to determine its feasibility for use in clinical practice. To this end, the nomogram was validated in the validation dataset by performing DCA. This innovative method provides an interpretation of medical outcomes in terms of the threshold probability to derive the net benefit, which is defined as the true-positive rate minus the false-positive rate weighted by the relative harm of false-positive and false-negative results (42). According to the decision curve, there was considerable overlap in the net benefits of the two models, which indicates that the addition of BRAF V600E mutation status to the nomogram did not improve its performance in terms of reclassification or integrated discrimination. These results reveal that the simpler US-based nomogram is more suitable for evaluating net benefit than model 2. When the threshold probability for the US-based nomogram is between 8% and 82%, it can be used to predict CLNM and would be more effective than a treat-none or treat-all strategy. The threshold probability for the US-based CLNM status is between 37% and 80%; therefore, the US-based nomogram has a greater net benefit and is more clinically feasible.

This study has some limitations. First, although external validation minimized the overfitting effects, the prediction performance was still affected by the retrospective design of the study. Furthermore, not all the patients in the study had undergone lateral neck dissection, and those lateral LNs without lateral neck dissection were considered to be negative. Finally, the study data were obtained from a single center. Therefore, while this study offers a preliminary indication that combining US features with clinical data can predict CLNM status, prospective multicenter studies are needed to validate the prediction nomogram.

In conclusion, we developed a US-based nomogram that integrates US characteristics with clinical features. The nomogram can be expediently applied to allow personalized preoperative prediction of CLNM in patients with PTC and tailoring of CCND accordingly.

Acknowledgments

We thank Zhiqiang Nie for help with the statistical analyses, and Thomas Barbour for help with English.

Funding: This study was supported by Guangzhou Municipal Science and Technology Planning Project (CN) (No. 201804010105, 202002030235).

Footnote

Reporting Checklist: The authors have completed the STROBE reporting checklist. Available at http://dx.doi.org/10.21037/gs-20-75

Data Sharing Statement: Available at http://dx.doi.org/10.21037/gs-20-75

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/gs-20-75). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013). The study was approved by the Guangdong Provincial People’s Hospital Research Ethics Committee (No. GDREC2018315H) and individual consent for this retrospective analysis was waived.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Siegel RL, Miller KD, Jemal A. Cancer statistics, 2019. CA Cancer J Clin 2019;69:7-34. [Crossref] [PubMed]
2. Sun W, Lan X, Zhang H, et al. Risk Factors for Central Lymph Node Metastasis in CN0 Papillary Thyroid Carcinoma: A Systematic Review and Meta-Analysis. PLoS One 2015;10:e0139021. [Crossref] [PubMed]
3. Baek SK, Jung KY, Kang SM, et al. Clinical risk factors associated with cervical lymph node recurrence in papillary thyroid carcinoma. Thyroid 2010;20:147-52. [Crossref] [PubMed]
4. Machens A, Hinze R, Thomusch O, et al. Pattern of nodal metastasis for primary and reoperative thyroid cancer. World J Surg 2002;26:22-8. [Crossref] [PubMed]
5. Liang K, He L, Dong W, et al. Risk factors of central lymph node metastasis in cN0 papillary thyroid carcinoma: a study of 529 patients. Med Sci Monit 2014;20:807-11. [Crossref] [PubMed]
6. Filetti S, Durante C, Hartl D, et al. Thyroid cancer: ESMO Clinical Practice Guidelines for diagnosis, treatment and follow-up. Ann Oncol 2019;30:1856-83. [Crossref] [PubMed]
7. Haugen BR, Alexander EK, Bible KC, et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 2016;26:1-133. [Crossref] [PubMed]
8. Perros P, Boelaert K, Colley S, et al. Guidelines for the management of thyroid cancer. Clin Endocrinol (Oxf) 2014;81 Suppl 1:1-122. [Crossref] [PubMed]
9. Takami H, Ito Y, Okamoto T, et al. Therapeutic strategy for differentiated thyroid carcinoma in Japan based on a newly established guideline managed by Japanese Society of Thyroid Surgeons and Japanese Association of Endocrine Surgeons. World J Surg 2011;35:111-21. [Crossref] [PubMed]
10. Choi JS, Kim J, Kwak JY, et al. Preoperative staging of papillary thyroid carcinoma: comparison of ultrasound imaging and CT. AJR Am J Roentgenol 2009;193:871-8. [Crossref] [PubMed]
11. Choi YJ, Yun JS, Kook SH, et al. Clinical and imaging assessment of cervical lymph node metastasis in papillary thyroid carcinomas. World J Surg 2010;34:1494-9. [Crossref] [PubMed]
12. Hwang HS, Orloff LA. Efficacy of preoperative neck ultrasound in the detection of cervical lymph node metastasis from thyroid cancer. Laryngoscope 2011;121:487-91. [Crossref] [PubMed]
13. Khokhar MT, Day KM, Sangal RB, et al. Preoperative High-Resolution Ultrasound for the Assessment of Malignant Central Compartment Lymph Nodes in Papillary Thyroid Cancer. Thyroid 2015;25:1351-4. [Crossref] [PubMed]
14. Viola D, Materazzi G, Valerio L, et al. Prophylactic central compartment lymph node dissection in papillary thyroid carcinoma: clinical implications derived from the first prospective randomized controlled single institution study. J Clin Endocrinol Metab 2015;100:1316-24. [Crossref] [PubMed]
15. Kim TH, Park YJ, Lim JA, et al. The association of the BRAF(V600E) mutation with prognostic factors and poor clinical outcome in papillary thyroid cancer: a meta-analysis. Cancer 2012;118:1764-73. [Crossref] [PubMed]
16. Li C, Lee KC, Schneider EB, et al. BRAF V600E mutation and its association with clinicopathological features of papillary thyroid cancer: a meta-analysis. J Clin Endocrinol Metab 2012;97:4559-70. [Crossref] [PubMed]
17. Xing M. BRAF mutation in papillary thyroid cancer: pathogenic role, molecular bases, and clinical implications. Endocr Rev 2007;28:742-62. [Crossref] [PubMed]
18. Xing M, Alzahrani AS, Carson KA, et al. Association between BRAF V600E mutation and recurrence of papillary thyroid cancer. J Clin Oncol 2015;33:42-50. [Crossref] [PubMed]
19. Chen BD, Zhang Z, Wang KK, et al. A multivariable model of BRAF and ultrasonographic features for predicting the risk of central lymph node metastasis in cN0 papillary thyroid microcarcinoma. Cancer Manag Res 2019;11:7211-7. [Crossref] [PubMed]
20. Howell GM, Nikiforova MN, Carty SE, et al. BRAF V600E mutation independently predicts central compartment lymph node metastasis in patients with papillary thyroid cancer. Ann Surg Oncol 2013;20:47-52. [Crossref] [PubMed]
21. Dong SY, Zeng RC, Jin LP, et al. BRAF mutation is not associated with central lymph node metastasis in all patients with papillary thyroid cancer: Different histological subtypes and preoperative lymph node status should be taken into account. Oncol Lett 2017;14:4122-34. [Crossref] [PubMed]
22. Kim SK, Chai YJ, Park I, et al. Nomogram for predicting central node metastasis in papillary thyroid carcinoma. J Surg Oncol 2017;115:266-72. [Crossref] [PubMed]
23. Lee KC, Li C, Schneider EB, et al. Is BRAF mutation associated with lymph node metastasis in patients with papillary thyroid cancer? Surgery 2012;152:977-83. [Crossref] [PubMed]
24. Chen J, Li XL, Zhao CK, et al. Conventional Ultrasound, Immunohistochemical Factors and BRAF Mutation in Predicting Central Cervical Lymph Node Metastasis of Papillary Thyroid Carcinoma. Ultrasound Med Biol 2018;44:2296-306. [Crossref] [PubMed]
25. Liu W, Cheng R, Ma Y, et al. Establishment and validation of the scoring system for preoperative prediction of central lymph node metastasis in papillary thyroid carcinoma. Sci Rep 2018;8:6962. [Crossref] [PubMed]
26. Wei X, Wang M, Wang X, et al. Prediction of cervical lymph node metastases in papillary thyroid microcarcinoma by sonographic features of the primary site. Cancer Biol Med 2019;16:587-94. [PubMed]
27. Xu JM, Xu XH, Xu HX, et al. Prediction of cervical lymph node metastasis in patients with papillary thyroid cancer using combined conventional ultrasound, strain elastography, and acoustic radiation force impulse (ARFI) elastography. Eur Radiol 2016;26:2611-22. [Crossref] [PubMed]
28. Balachandran VP, Gonen M, Smith JJ, et al. Nomograms in oncology: more than meets the eye. Lancet Oncol 2015;16:e173-80. [Crossref] [PubMed]
29. Hei H, Song Y, Qin J. A nomogram predicting contralateral central neck lymph node metastasis for papillary thyroid carcinoma. J Surg Oncol 2016;114:703-7. [Crossref] [PubMed]
30. Huang YQ, Liang CH, He L, et al. Development and Validation of a Radiomics Nomogram for Preoperative Prediction of Lymph Node Metastasis in Colorectal Cancer. J Clin Oncol 2016;34:2157-64. [Crossref] [PubMed]
31. Grant EG, Tessler FN, Hoang JK, et al. Thyroid Ultrasound Reporting Lexicon: White Paper of the ACR Thyroid Imaging, Reporting and Data System (TIRADS) Committee. J Am Coll Radiol 2015;12:1272-9. [Crossref] [PubMed]
32. Kwak JY, Han KH, Yoon JH, et al. Thyroid imaging reporting and data system for US features of nodules: a step in establishing better stratification of cancer risk. Radiology 2011;260:892-9. [Crossref] [PubMed]
33. Vickers AJ, Cronin AM, Elkin EB, et al. Extensions to decision curve analysis, a novel method for evaluating diagnostic tests, prediction models and molecular markers. BMC Med Inform Decis Mak 2008;8:53. [Crossref] [PubMed]
34. Cantisani V, Grazhdani H, Drakonaki E, et al. Strain US Elastography for the Characterization of Thyroid Nodules: Advantages and Limitation. Int J Endocrinol 2015;2015:908575. [Crossref] [PubMed]
35. Sidhu PS, Cantisani V, Dietrich CF, et al. The EFSUMB Guidelines and Recommendations for the Clinical Practice of Contrast-Enhanced Ultrasound (CEUS) in Non-Hepatic Applications: Update 2017 (Short Version). Ultraschall in der Medizin 2018;39:154-80. [Crossref] [PubMed]
36. Săftoiu A, Gilja OH, Sidhu PS, et al. The EFSUMB Guidelines and Recommendations for the Clinical Practice of Elastography in Non-Hepatic Applications: Update 2018. Ultraschall in der Medizin 2019;40:425-53. [Crossref] [PubMed]
37. Zhao H, Li H. Meta-analysis of ultrasound for cervical lymph nodes in papillary thyroid cancer: Diagnosis of central and lateral compartment nodal metastases. Eur J Radiol 2019;112:14-21. [Crossref] [PubMed]
38. Collins GS, Reitsma JB, Altman DG, et al. Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement. BMJ 2015;350:g7594. [Crossref] [PubMed]
39. Jinih M, Foley N, Osho O, et al. BRAF mutation as a predictor of thyroid malignancy in indeterminate nodules: A systematic review and meta-analysis. Eur J Surg Oncol 2017;43:1219-27. [Crossref] [PubMed]
40. Paek SH, Kim BS, Kang KH, et al. False-negative BRAF V600E mutation results on fine-needle aspiration cytology of papillary thyroid carcinoma. World J Surg Oncol 2017;15:202. [Crossref] [PubMed]
41. Lang BH, Chai YJ, Cowling BJ, et al. Is BRAFV600E mutation a marker for central nodal metastasis in small papillary thyroid carcinoma? Endocr Relat Cancer 2014;21:285-95. [Crossref] [PubMed]
42. Van Calster B, Vickers AJ. Calibration of risk prediction models: impact on decision-analytic performance. Med Decis Making 2015;35:162-9. [Crossref] [PubMed]