Development and validation of the nomogram for predicting preoperative vocal cord palsy in thyroid cancer patients

Introduction

Thyroid cancer is the most common malignancy of the endocrine system in China and other countries (1-4). Thyroidectomy is considered the primary intervention and low-risk operation for the diagnosis and treatment of patients with thyroid cancer (5,6). However, for the head and neck surgeons, it should be noted that vocal cord palsy (VCP) and even rapidly progressive trachea asphyxia caused by the injury of the recurrent laryngeal nerve (RLN) are severe complications after thyroid surgery (7,8), leading to a decline in quality of life and even a medical emergency. Therefore, preoperative vocal cord assessment is necessary to prevent RLN injury during the perioperative period. It is the basis for baseline examination of the vocal cords, and is also one of the invasive evidences for malignant thyroid tumors (9).

2015 American Thyroid Association (ATA) management guidelines for adult patients with thyroid nodules and differentiated thyroid cancer recommended that doctors conduct a routine assessment of voice and selective laryngoscopy in high-risk patients (10). The NCCN (National Comprehensive Cancer Network) clinical practice guidelines in oncology suggested that preoperative laryngoscopy could be based on cervical surgery history, abnormal voice status, central lymph node involvement, or invasive diseases (11). However, in most hospitals in China, otolaryngologists usually performed laryngoscopy one week before the operation for all patients with suspected thyroid cancer. The incidence of preoperative VCP is relatively low (1–6%) (12,13). Non-Selective laryngoscopy may result in a large number of unnecessary examinations and medical costs. Meanwhile, many preoperative clinical factors, such as neck surgery history, voice complaints, invasive thyroid cancer, are considered highly relevant for preoperative VCP (14). Therefore, routine laryngoscopy seems to be controversial (15). There were many studies to explore the risk factors of preoperative VCP. However, the research on the prediction model of VCP occurrence has not been completed.

Our study retrospectively analyzed the incidence of preoperative VCP in thyroid cancer patients in Union Hospital, Tongji Medical College of Huazhong University of Science and Technology (Wuhan Union Hospital), intending to identify clinical indicators of preoperative VCP. At the same time, we established a visual nomogram based on preoperative independent risk features to predict the possibility of VCP accurately. We verified the nomograph’s performance in discrimination and accuracy using the internal cohort and external cohort from prospective multi-center thyroid cancer research in China.

We present the following article in accordance with the TRIPOD reporting checklist (available at http://dx.doi.org/10.21037/gs-20-621).

Methods

Study cohort and patients

We performed a retrospective study recruiting patients from the thyroid and breast disease center database in Wuhan Union Hospital from January 2008 to August 2019. The external cohort was from differentiated thyroid cancer in China (DTCC) study, which is the most considerable prospective multi-center research in China, including nine large thyroid cancer research centers. The trial was registered at ClinicalTrials.gov (NCT02638077). Variables and outcomes in all patients diagnosed thyroid cancer were reviewed including age, gender, BMI, complaints of voice, history of neck surgery (included previous thyroid surgery), maximum ultrasound nodule size, cervical lymph node report, TSH, FT3, FT4, and laryngoscopy results. Inclusion criterions included thyroid cancer patients who underwent a preoperative vocal cord assessment by fiberoptic laryngoscopy. VCP was defined as a reduced or absent movement in one or more vocal cords (VCs). A voice symptom only referred to hoarseness complained by patients or discovered by the first visited physician and finally recorded after assessment by experienced surgeons. Suspicious lymph node metastasis (LNM) (clinical N1, cN1) included N1a (central nodal metastasis) or (and) N1b (lateral nodal metastasis) (clinical N1, cN1). And they were enlarged, exhibited a loss of fatty hilum, were more round rather than oval in shape, and showed hyper echogenicity, cystic change, calcifications, and peripheral vascularity. Surgical treatment included total thyroidectomy and thyroid lobectomy. The laterality to be prior removed is located on the side diagnosed thyroid cancer.

Construction and validation of the nomogram

In order to determine possible predictors for preoperative VCP, patients’ baseline characteristics, ultrasonographic results of tumor and neck lymph nodes, preoperative biochemical parameters, and fine needle aspiration cytology (FNAC) were compared between VCP group and NP group (no palsy) by Chi-squared tests and t-test. Univariate analysis and multivariate logistic regression were used for the analysis of risk factors. Predicting factors with P<0.1 in univariate analysis were used as candidate risk factors for the stepwise multivariate logistic regression model with backward selection. The multivariate proportional hazards model was used to identify independent factors associated with preoperative vocal palsy in the malignant thyroid tumor group. All significant independent factors in multivariate logistic regression were incorporated in a nomogram and, each predictor was assigned to a point. By summing each variable’s scores ranging from 0 to 100, the probability of VCP in a specific patient was predicted. We divided patients in 2013–2019 into training and internal validation sets according to admission time (2,338 patients from 2013–2017 and 485 patients from 2018–2019). The concordance index (C-index) was provided to estimate the nomogram’s discrimination performance in all training, internal and external validation sets. A calibration method with bootstrapping was utilized in three sets to visualize the variance between the predicted and actual probabilities. Meanwhile, we further evaluated the effectiveness of predicting VCP using ROC curves in these cohorts. The value of AUC ranged from 0.5 (no discrimination) to 1.0 (perfect discrimination). We also compared the diagnostic accuracy of VCP, including sensitivity (Se), specificity (Sp), positive predictive value (PPV) and negative predictive value (NPV) in the development cohorts and validation cohorts. In univariable and multivariable analysis, missing data will be separated as a single cohort. In the processing of nomogram construction, we did not include the missing data into the scoring system.

Statistical analysis

All statistical analyses were conducted using SPSS version 23.0 (SPSS, Inc., Chicago, IL, USA) or R software version 3.6.2 (http://www.r-project.org). Linear and categorical variables were presented as mean (standard deviation) and number (frequency), respectively. Variables were compared by the χ² test or Student’s t-test as appropriate. All P values were two-sided, and a value of P<0.05 was considered as statistically significant.

Ethical statement

The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013) and approved by Ethical Committee of the Union Hospital, Tongji Medical College of Huazhong University of Science and Technology (No. 0304-01). The requirement of informed consent for publication of data was not required for the retrospective nature of the study.

Results

Patient characteristics

The clinical variables in the development and validation cohort were shown in Table 1. A total of 5,354 malignant thyroid tumor patients were identified at Wuhan Union Hospital between 2008 and 2019. There were 84 (1.57%) preoperative VCP patients and 5,270 (98.43%) patients without VCP (Table 1). The distribution of VCP patients significantly varied with gender (female: 65.48% versus 77.27%; P=0.011) and age for both the groups. Patients in the VCP group were significantly older than patients in the NP group, ranging from 35 to 89 years, with a mean of 49±12.4 years (P<0.001; Table 1). Furthermore, VCP patients showed higher BMI (24.1±3.1 vs. 23.3±3.4; P=0.022) and tumor diameter (2.2±1.6 versus 1.5±1.1; P<0.001) than NP patients and a significantly higher proportion of patients complained about voice symptoms compared to the NP group (5.95% versus 0.17%; P<0.001). The proportion of neck surgery history in the VCP group was nearly seven times higher than that of adults with NP (55.95% versus 7.59%; P<0.001). Meanwhile, Table 1 shown that a higher proportion of suspicious lymph nodes were found (58.33% versus 39.92%, P<0.01). No significant differences were found when analyzing fine needle aspiration cytology (FNAC) history (P=0.223) and preoperative biochemical parameters, including TSH (P=0.206), FT3 (P=0.412) and FT4 (P=0.072). Simultaneously, a total of 1,225 thyroid cancer patients undergoing laryngoscopy were included in the external validation cohort from the DTCC database with an average age of 41.62±11.60, including 348 male and 877 female patients. There were 21 patients diagnosed with preoperative VCP (1.71%), 517 patients underwent FNAC and only four patients with neck surgery history in DTCC database.

Table 1 Baseline characteristics and bio-chemical parameters in development cohort and validation cohort
Full table

Analysis of high-risk factors of VCP

After adjusting confounding factors including age, sex, largest tumor size, BMI, history of neck surgery, voice complaints, neck lymph nodes, age at diagnosis (OR: 1.04; P=0.006), history of neck surgery (OR: 11.57; P<0.001), voice symptoms (OR: 32.75; P<0.001), tumor diameter (OR: 1.04; P<0.001) and suspicious lymph nodes (clinical N1) (OR: 3.25; P<0.001) were identified as being independently associated with VCP in multivariate analyses (Table 2). However, sex and BMI were not associated with VCP (P>0.05).

Table 2 Multivariate logistic analysis of risk factors for preoperative vocal cord palsy in thyroid cancer patients
Full table

Novel nomogram and validation

We established a nomogram that incorporated significant predictive factors including age, tumor size, history of neck surgery, voice complaints and neck lymph nodes from multivariate analysis (Figure 1). The risk of VCP predicted by our nomogram ranged from 0.01 to 0.8. After matching each patient parameter result on their corresponding axis and calculating the respective points, surgeons could determine VCP presence probabilities by adding the total points from all parameters.

Figure 1 Nomogram to discriminate VCP from thyroid cancer patients. VCP, vocal cord palsy.

In the construction cohort from Wuhan Union hospital, discrimination of nomogram was found to be excellent with a concordance index (CI) of 0.86 in the training and 0.83 internal validation sets, respectively. Bias-corrected and ideal curves showed that the nomogram internally had a good fit (Figure 2A,B). In the external validation cohort from the DTCC database, the data set was from different thyroid cancer centers distributed throughout China, the nomogram was also well-calibrated in the validation cohort (Figure 2C). And nomogram also indicated a satisfactory discriminative capacity for predicting VCP (AUCs were 0.856 in the training set and 0.840 in the internal validating sets respectively) (Figure 3A,B). In DTCC database, the AUC of the nomogram increased to 0.895 (Figure 3C).

Figure 2 The calibration curve for predicting the occurrence of VCP in (A) training set (B), internal validation set, (C) external validation set. VCP, vocal cord palsy.

Figure 3 ROC curves in (A) training groups, (B) internal validating groups, (C) external validating groups for validating nomogram model. In the training set, the AUC of nomogram was 0.856. In the internal and external validation set, the AUC was 0.840 and 0.895 respectively. ROC, receiver operating characteristic; AUC, area under curve.

Meanwhile, as demonstrated in Table 3, under the cut-off value of 94.7 in the training cohort, the sensitivity, specificity, positive predictive value and negative predictive value of the nomogram for differential diagnosis of VCP were 74.29%, 80.03%, 5.35% and 99.51%, respectively. In the internal validation cohort and external validation cohort, these values were 76.92%, 86.65%, 13.70%, 99.27%, respectively and 80.95%, 80.07%, 6.72% and 99.58%, respectively.

Table 3 Diagnostic accuracy of the nomogram for estimating the presence of VCP
Full table

Discussion

The necessity of routine preoperative laryngeal examinations remains controversial. The most important reason for this recommendation includes a better assessment of preoperative vocal cord states (16). The VCP is an essential indicator of RLN injury caused by tumor extrathyroidal invasion (17,18). However, the incidence of preoperative VCP is very low and some patients diagnosed with VCP before surgery recovered well after surgery (14,19). Most clinicians recommend routine preoperative laryngoscopy for patients with no selection in China, ignoring the risk factors highly associated with preoperative VCP. These risk factors can be used to identify high-risk patients who actually need laryngoscopy accurately. In addition, regardless of whether a laryngoscopy is performed before surgery, the protection of RLN is essential for thyroid surgery unless the tumor is invaded locally and the nerves cannot be separated. Thus, there is an urgent need to establish an accurate prediction model to guide clinicians to perform pre-laryngoscopy on patients. Many studied have reported that advanced tumors, hoarseness and neck surgery history are related to preoperative VCP and simulated the process of selective laryngoscopy according to the risk factors (18,20). All these factors have not been integrated into a quantified model and verified from a multi-center external cohort. That is why we have developed a simplified VCP hierarchical scoring system, which can calculate the cumulative VCP risk and determine the preoperative laryngeal examination. Through internal and external verification, it has an overall diagnostic performance.

In this study, the incidence of preoperative VCP was 1.57%, among 5,354 patients diagnosed with thyroid cancer in the thyroid study center of Wuhan Union hospital, which is comparable with 1.77% among DTCC, and slightly higher than 0.68% shown by Maher (21) and 0.6% shown by Goretzki et al. (22). Patients in DTCC came from hospitals with high-volume thyroid cancer surgery, which included a higher proportion of invasive thyroid cancer patients, such as local invasion or LNM. However, they did not increase the risk of VCP. Other studies reported higher rates of preoperative palsy (12,16). This difference in the incidence of preoperative VCP reveals variations existing in baseline demographic data of included patients, such as the severity of thyroid cancer, voice complaints and neck surgery history. The latter is a potential etiology of injury of RLN (23,24). It is important to note that the hoarseness rate of VCP patients was only 5.95% in our database, and the incidence of external cohort was only 1.31%. One reason for the low proportion of hoarseness in patients with thyroid cancer is that patients with cough or hoarseness were required to delay admission almost a week to decrease the anesthesia risk. Meanwhile, we also noticed that the hoarseness’s incidence rate was significantly different between complained by the patients and mainly determined by doctors (12).

In our study, we found that independent risk factors of VCP included age, voice complaints, hoarseness, previous neck history, tumor size, suspicious neck LNM by ultrasound. Similar to previous studies, the history of FNA (fine-needle aspiration) also cannot be used as an indicator of VCP, which reveals that FNA does not affect RLN function (21). Meanwhile, voice symptoms, large nodules, and a history of neck surgery had also been found positively correlated with VCP before our studies (12,25). They had been even suggested as a direct indication for preoperative laryngoscopy (21). Nevertheless, not all VCP patients were found to have these characteristics. In our study, only 49 patients (58.33%) with VCP were identified to have voice complaints or a history of neck surgery, implying that compensation from contralateral cord function may overshadow voice symptoms (10). On the other hand, there is a significant difference in age between the VCP and NP patients, which was also a risk factor for the VCP in the cohort of early investigations (17,21). However, it is not considered as an independent indicator of selective laryngoscopy due to overlapping with other risk factors (21). In our results, age was regarded as a fundamental parameter in the nomogram. However, there were still 3 VCP patients (6.25%) with no other risk factors except for older age (≥55 years old), and the total score does not reach the threshold value (Table S1). Stage of cN1 usually indicates the locally advanced thyroid cancer stage, which is also related to the more invasion and high relapse rate of the tumor (26,27). Therefore, we believe that suspicious lymph nodes under ultrasound are significant indications for laryngoscopy. However, the performance of detection of cN1 depended on the accuracy of ultrasound devices and the experiences of ultrasound doctors. Our results further increase the 2015 ATA guidelines’ recommendation level, which recommends preoperative laryngeal examination for patients with extensive central lymph node metastases.

Nomograms can promote individualized decision-making by providing accurate risk evaluation and discrimination characteristics. Under the nomogram, there are about 20% (Table 3: 486/2,338 =20.7%; 73/485 =15.1%; 254/1,205 =21.1%) patients with positive nomogram should be performed preoperative laryngoscopy. The nomogram identified low-risk patients and contributed to a decrease in quantity of performing this technique. Meanwhile, all the preoperative factors included in the nomogram are accessible and have the advantage of low costs. In our model, the cut-off value is 94.7, which implied that all patients with voice symptoms and most patients with neck surgery history should be examined (patients with voice complaints got 100 scores and patients with neck surgery history got almost 70 scores). Moreover, other indications of laryngeal examinations should at least include two high-risk factors. The c-statistic in internal verification using the 2018–2019 database is 0.83 and the external verification using the DTCC database is 0.85. Compared with the internal verification set, the AUC of the DTCC queue has improved, indicating that the nomogram has a higher ability to discriminate VCP. Calibration also showed that the nomogram has an acceptable prediction accuracy. Our work, for the first time, found and validated a new diagnostic model to predict preoperative VCP based on the largest thyroid cancer database in China, to support clinical decision-making. Not only will the nomogram help avoid unnecessary patient discomfort, but it also decreases medical costs.

However, there were still 20% false-negative patients in nomogram negative patients. Our model still was not able to include patients that did not exhibit suspicious high-risk factors. Mostly, it is easy to miss younger patients with small cancers near RLN. However, the magnitude of this risk to omit younger patients with small tumors remains inconclusive because the subgroup analysis for VCP in young patients is currently limited. Preoperative tumor localization of ultrasound and intraoperative nerve monitoring (INM) are considered key links in the management and prevention of nomogram negative patients. Preoperative ultrasound may play an important role in alerting surgeons of the status of RLN by indicating the extra-thyroid invasion, especially for small tumors which anatomic location were relatively backward. INM can be used in high-risk patients, which provides additional evidence for the RLN status, has a high negative predictive value, and has a significant heterogeneity in the therapeutic effect of RLN injury (28,29). Meanwhile, retrospective study bias may make some preoperative factors lose statistical significance.

Conclusions

We recommend that physicians consider the visualized nomogram model developed in this study before the thyroid cancer surgery, enabling patients and clinicians to quickly assess quantified information regarding the probability of VCP and avoid unnecessary examinations.

Acknowledgments

The authors would like to thank all clinicians and researchers contributing to patients Care, preoperative laryngeal examination as well as collecting data in the Union Hospital, Tongji Medical College, Huazhong University of Science and Technology.

Funding: None.

Footnote

Reporting Checklist: The authors have completed the TRIPOD reporting checklist. Available at http://dx.doi.org/10.21037/gs-20-621

Data Sharing Statement: Available at http://dx.doi.org/10.21037/gs-20-621

Peer Review File: Available at http://dx.doi.org/10.21037/gs-20-621

Conflicts of Interest: All authors have completed the ICMJE uniform disclosure form (available at http://dx.doi.org/10.21037/gs-20-621). The authors have no conflicts of interest to declare.

Ethical Statement: The authors are accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. The study was conducted in accordance with the Declaration of Helsinki (as revised in 2013) and approved by Ethical Committee of the Union Hospital, Tongji Medical College of Huazhong University of Science and Technology (No. 0304-01). The requirement of informed consent for publication of data was not required for the retrospective nature of the study.

Open Access Statement: This is an Open Access article distributed in accordance with the Creative Commons Attribution-NonCommercial-NoDerivs 4.0 International License (CC BY-NC-ND 4.0), which permits the non-commercial replication and distribution of the article with the strict proviso that no changes or edits are made and the original work is properly cited (including links to both the formal publication through the relevant DOI and the license). See: https://creativecommons.org/licenses/by-nc-nd/4.0/.

References

1. Zhao L, Pang P, Zang L, et al. Features and trends of thyroid cancer in patients with thyroidectomies in Beijing, China between 1994 and 2015: a retrospective study. BMJ Open 2019;9:e023334. [Crossref] [PubMed]
2. Du L, Wang Y, Sun X, et al. Thyroid cancer: trends in incidence, mortality and clinical-pathological patterns in Zhejiang Province, Southeast China. BMC Cancer 2018;18:291. [Crossref] [PubMed]
3. Chen AY, Jemal A, Ward EM. Increasing incidence of differentiated thyroid cancer in the United States, 1988-2005. Cancer 2009;115:3801-7. [Crossref] [PubMed]
4. Du L, Li R, Ge M, et al. Incidence and mortality of thyroid cancer in China, 2008-2012. Chin J Cancer Res 2019;31:144-51. [Crossref] [PubMed]
5. Khadra H, Mohamed S, Hauch A, et al. Safety of same-day thyroidectomy: meta-analysis and systematic review. Gland Surg 2017;6:292-301. [Crossref] [PubMed]
6. Yang Y, Huang K, Huang Y, et al. Assessment of the safety and feasibility of 24-hour hospitalization after thyroidectomy. Can J Physiol Pharmacol 2018;96:893-7. [Crossref] [PubMed]
7. Bergenfelz A, Jansson S, Kristoffersson A, et al. Complications to thyroid surgery: results as reported in a database from a multicenter audit comprising 3,660 patients. Langenbecks Arch Surg 2008;393:667-73. [Crossref] [PubMed]
8. Rosato L, Avenia N, Bernante P, et al. Complications of thyroid surgery: analysis of a multicentric study on 14,934 patients operated on in Italy over 5 years. World J Surg 2004;28:271-6. [Crossref] [PubMed]
9. Nam IC, Bae JS, Shim MR, et al. The importance of preoperative laryngeal examination before thyroidectomy and the usefulness of a voice questionnaire in screening. World J Surg 2012;36:303-9. [Crossref] [PubMed]
10. Haugen BR, Alexander EK, Bible KC, et al. 2015 American Thyroid Association Management Guidelines for Adult Patients with Thyroid Nodules and Differentiated Thyroid Cancer: The American Thyroid Association Guidelines Task Force on Thyroid Nodules and Differentiated Thyroid Cancer. Thyroid 2016;26:1-133. [Crossref] [PubMed]
11. Haddad RI, LWM, Bischoff L, et al. NCCN clinical practive guidelines in oncology (NCCN guidelines): thyroid carcinoma. National Comprehensive Cancer Network; version 1.2019. Available online: https://www.nccn.org/professionals/physician_gls/pdf/thyroid.pdf, accessed March 28, 2019.
12. Lee CY, Long KL, Eldridge RJ, et al. Preoperative laryngoscopy in thyroid surgery: Do patients’ subjective voice complaints matter? Surgery 2014;156:1477-82; discussion 1482-3. [Crossref] [PubMed]
13. Wang CC, Wang CP, Tsai TL, et al. The basis of preoperative vocal fold paralysis in a series of patients undergoing thyroid surgery: the preponderance of benign thyroid disease. Thyroid 2011;21:867-72. [Crossref] [PubMed]
14. Franch-Arcas G, Gonzalez-Sanchez C, Aguilera-Molina YY, et al. Is there a case for selective, rather than routine, preoperative laryngoscopy in thyroid surgery? Gland Surg 2015;4:8-18. [PubMed]
15. Hodin R, Clark O, Doherty G, et al. Voice issues and laryngoscopy in thyroid surgery patients. Surgery 2013;154:46-7. [Crossref] [PubMed]
16. Farrag TY, Samlan RA, Lin FR, et al. The utility of evaluating true vocal fold motion before thyroid surgery. Laryngoscope 2006;116:235-8. [Crossref] [PubMed]
17. Chiang FY, Lin JC, Lee KW, et al. Thyroid tumors with preoperative recurrent laryngeal nerve palsy: clinicopathologic features and treatment outcome. Surgery 2006;140:413-7. [Crossref] [PubMed]
18. Randolph GW, Kamani D. The importance of preoperative laryngoscopy in patients undergoing thyroidectomy: voice, vocal cord function, and the preoperative detection of invasive thyroid malignancy. Surgery 2006;139:357-62. [Crossref] [PubMed]
19. Lang BH, Chu KK, Tsang RK, et al. Evaluating the incidence, clinical significance and predictors for vocal cord palsy and incidental laryngopharyngeal conditions before elective thyroidectomy: is there a case for routine laryngoscopic examination? World J Surg 2014;38:385-91. [Crossref] [PubMed]
20. Myssiorek D. Recurrent laryngeal nerve paralysis: anatomy and etiology. Otolaryngol Clin North Am 2004;37:25-44. v. [Crossref] [PubMed]
21. Maher DI, Goare S, Forrest E, et al. Routine Preoperative Laryngoscopy for Thyroid Surgery Is Not Necessary Without Risk Factors. Thyroid 2019;29:1646-52. [Crossref] [PubMed]
22. Goretzki PE, Schwarz K, Brinkmann J, et al. The impact of intraoperative neuromonitoring (IONM) on surgical strategy in bilateral thyroid diseases: is it worth the effort? World J Surg 2010;34:1274-84. [Crossref] [PubMed]
23. Rosenthal LHS, Benninger MS, Deeb RH. Vocal fold immobility: a longitudinal analysis of etiology over 20 years. Laryngoscope 2007;117:1864-70. [Crossref] [PubMed]
24. Kriskovich MD, Apfelbaum RI, Haller JR. Vocal fold paralysis after anterior cervical spine surgery: incidence, mechanism, and prevention of injury. Laryngoscope 2000;110:1467-73. [Crossref] [PubMed]
25. Ahmad S, Muzamil A, Lateef M. A Study of incidence and etiopathology of vocal cord paralysis. Indian J Otolaryngol Head Neck Surg 2002;54:294-6. [PubMed]
26. Podnos YD, Smith D, Wagman LD, et al. The implication of lymph node metastasis on survival in patients with well-differentiated thyroid cancer. Am Surg 2005;71:731-4. [Crossref] [PubMed]
27. Zhang J, Cheng X, Shen L, et al. The Association Between Lymph Node Stage and Clinical Prognosis in Thyroid Cancer. Front Endocrinol (Lausanne) 2020;11:90. [Crossref] [PubMed]
28. Barczyński M, Konturek A, Stopa M, et al. Randomized controlled trial of visualization versus neuromonitoring of the external branch of the superior laryngeal nerve during thyroidectomy. World J Surg 2012;36:1340-7. [Crossref] [PubMed]
29. Barczyński M, Konturek A, Cichon S. Randomized clinical trial of visualization versus neuromonitoring of recurrent laryngeal nerves during thyroidectomy. Br J Surg 2009;96:240-6. [Crossref] [PubMed]