Machine learning methods in differential diagnosis of ACTH-dependent hypercortisolism

BACKGROUND

Endogenous hypercortisolism is an extremely rare and one of the most severe endocrine diseases caused by long-term body exposure to excessive levels of cortisol. To avoid the development of life-threatening complications and patient disability, timely diagnosis of this condition is necessary to establish the disease ethiology and subsequent adequate treatment [1]. ACTH-secreting pituitary adenomas (Cushing disease) account for up to 80% of nosologies leading to hypercortisolism. A much rarer type of ACTH-dependent hypercortisolism is ectopic ACTH syndrome (EAS) caused by overproduction of ACTH, or, less frequently, overproduction of corticotropin-releasing hormone (CRH) or neuroendocrine tumour (NET) of various localisations [2].

Differentiating between Cushing disease and EAS is very challenging, especially in patients without adenoma visualisation or with pituitary microadenomas less than 6 mm in diameter and/or negative results of topical NET diagnostics obtained through radiological, functional, and receptor imaging [3].

The gold standard for differential diagnosis of ACTH-dependent forms of endogenous hypercortisolism is bilateral selective blood sampling from the inferior stony sinuses with determination of the ACTH centre/periphery gradient before and after administration of the stimulating agent [4]. Since its introduction, the method has undergone significant changes and improvements to enhance diagnostic accuracy [5]. The results of a study conducted in Russia’s Endocrinology Research Centre indicate high sensitivity (98.31% (95% CI 95.16–99.43)) and specificity (94.64% (95% CI 85.39–98.16)) of selective blood sampling using desmopressin stimulation [6].

Despite the high diagnostic accuracy of the method, selective blood sampling from the inferior stony sinuses can be performed in only a handful of Russia’s medical institutions due to the technical difficulties of this procedure, high cost, high man-hour expenses and potential complications of this invasive intervention. In turn, non-invasive diagnostic tests used for differential diagnosis, such as the high-dose dexamethasone test (HDT) and peripheral corticoliberin test, do not have sufficient sensitivity and specificity [7]. To improve the differential diagnosis of ACTH-dependent forms of endogenous hypercortisolism, there is a clear need for more accessible, easy-to-use and non-invasive methods that can be used in conditions where selective blood sampling from the lower stony sinuses is impossible.

Development of digital technologies and a trend towards personalised medicine in recent years have encouraged active implementation of artificial intelligence methods to support physician’s decision-making in many areas of medicine [8]. Machine learning methods are considered to be the main tools of artificial intelligence and have been increasingly used in diagnostic and predictive studies. To enable early and differential diagnosis of ACTH-dependent hypercortisolism, an algorithm combining the analysis of simple medical history data with the results of clinical and instrumental examination can be proposed as a non-invasive screening or primary diagnosis method.

PURPOSE

To develop a non-invasive method of differential diagnosis of ACTH-dependent endogenous hypercortisolism based on data accessible in regular clinical practice and to evaluate the effectiveness of an optimal algorithm for predicting the probability of ectopic ACTH syndrome developed through machine learning methods.

MATERIALS AND METHODS

Study design

A single-centre cross-sectional cohort study with a retrospective data analysis. The study included patients observed and/or treated in Russia’s Endocrinology Research Centre from 28 October 2015 to 29 December 2022.

Inclusion criteria

Inclusion criteria: patients of both sexes with ACTH-dependent endogenous hypercortisolism confirmed as per federal clinical guidelines [7] by at least two laboratory tests: elevated 24-h urinary free cortisol and/or late-night salivary cortisol or late-night blood cortisol and/or negative small-dose (1 mg) dexamethasone test with (cut-off point: 50 nmol/L); morning ACTH concentration ≥10 pg/ml; no visualisation of pituitary adenoma on MRI or pituitary adenoma size less than 10 mm.

Exclusion criteria: patients with other verified forms of hypercortisolism; a history of bilateral adrenalectomy; taking medications for conservative treatment of hypercortisolism; therapy with prolonged-acting somatostatin analogues; women during pregnancy, childbirth, or breastfeeding; persons suffering from psychiatric diseases; patients with poorly maintained medical records; patients with missing values of the analysed indicators.

Population sampling

To build a model of differential diagnosis between Cushing disease and EAS, we used data from a sample of ACTH-dependent endogenous hypercortisolism patients selected from the general cohort of patients who underwent in-patient examination and neurosurgical treatment at the Russia’s Endocrinology Research Centre or underwent surgical treatment of NET with ectopic ACTH production in a relevant specialised medical institution in 2015–2022.

Medical intervention procedures

In all patients included in the study, ACTH-dependent endogenous hypercortisolism was laboratory and clinically confirmed at the screening stage in accordance with clinical guidelines [7]. In order to differentiate between Cushing disease and EAS, 113 patients included in the study underwent HDT. A decrease in blood cortisol at 08:00 by more than 60% from baseline after 8 mg dexamethasone administration the day before was considered as a central form of endogenous hypercortisolism (Cushing disease), whereas a decrease in cortisol level by less than 60% was in favour of EAS.

All patients included in the study underwent selective blood sampling from the inferior cavernous sinuses with desmopressin stimulation (Ferring Pharmaceuticals Desmopressin Acetate 4 µg per 1 ml for intravenous, intramuscular and subcutaneous injections) at a 8 µg dose. Selective blood sampling results were evaluated based on the estimated maximum ratio of sinus ACTH to peripheral ACTH before desmopressin stimulation and after IV administration of the drug. Any ACTH centre/periphery gradient ≥2 before desmopressin stimulation and/or ≥3 thereafter was considered to be in favour of Cushing disease, whereas lower values of the ACTH centre/periphery gradient were considered as EAS [9].

Measurement equipment

Brain MRI was performed with Magnetom Harmony magnetic resonance tomograph (Siemens, Germany) with field strength from 1.5 to 3 Tesla and gadolinium contrast agent.

Hormonal ACTH measurements (reference interval: morning 7.2–63.3 pg/ml, evening 2–25.5 pg/ml), late night serum cortisol measurements (64–327 nmol/L), late night salivary cortisol (0.5–9.6 nmol/L) were conducted by electrochemiluminescent method on a Cobas 6000 Module e601 analyser (Roche); 24-h urinary free cortisol (100–379 nmol/day) was measured by immunochemiluminescent method on Vitros ECi device; potassium level (3.5–5.1 mmol/L) was measured with Architect c8000 analyser (Abbott, USA).

Modelling methods

The concept of developing and validating predictive models is presented in Figure 1.

Data pre-processing included removing features with near-zero variance and screening out collinear predictors. In addition, predictors were normalised and centred for those algorithms that required it. To avoid data leakage, all pre-processing and transformation was performed after splitting the dataset.

The sample split into the training set and the test set was performed using an 80/20 random stratified selection. In order to confirm the consistency of the data distribution in the training and test sets, analyses were performed by estimating the modified multivariate Mahalanobis distance (Eklavya Jain, 2022) and testing the null hypothesis of consistency of distributions (Monte Carlo test). The analysis showed a small distance of 12.7 between the training and the test sets. Accepting the null hypothesis with a p-value significance at 0.755, we concluded that this separation can be used to measure the model effectiveness.

We subjected the available data to mathematical processing to select an optimal set of indicators that would be more likely to be characteristic for Cushing disease or EAS. Moreover, we searched for parameters recommended by the software on the basis of mathematical calculations in order to exercise optimal control over the model training process. Baseline clinical features presented in Table 1 were considered as potential parameters.

Eleven machine learning algorithms were compared to develop an EAS predictive model for patients with ACTH-dependent endogenous hypercortisolism: Linear Discriminant Analysis, Logistic Regression, Elastic Network (GLMNET), Support Vector Machine (SVM Radial), k-Nearest Neighbours (kNN), Naïve Bayes, Binary Decision Tree (CART), C5.0 Decision Tree Algorithms, CART Bootstrap Aggregation (Bagged CART), Random Forest, and Stochastic Gradient Boosting (GBM). The result of the prediction is the probability of EAS in the patient, expressed as a percentage value.

Model training

To identify the best algorithm for predicting EAS among patients with ACTH-dependent endogenous hypercortisolism, nested cross-validation with five external and three internal cycles (3×5 Nested Cross-Validation) was used as per the scheme proposed by Isci S. et al. [10]. This approach provides a more accurate estimation of Generalisation error while optimising the hyperparameters and has shown its applicability in the subject domain.

The Boruta algorithm (Kursa and Rudnicki, 2010) was used to select predictors for model training. This algorithm is a wrapper-based marker selection method; it is tuned to find the minimum optimal marker set instead of all possible relevant markers, resulting in an unbiased and consistent selection of important predictors.

Training and performance evaluation of the machine learning algorithms were performed on a training dataset. After selecting the best algorithm, its final evaluation was performed on the test set.

Model performance assessment

Performance results of the models were compared with one another by analysing standard metrics derived from the confusion matrix: 1) Accuracy; 2) ROC-AUC (Area Under Receiver Operating Characteristic Curve; 3) Sensitivity; 4) Specificity; 5) Precision; 6) Recall; 7) F₁-measure.

The main metrics for selecting the final model were: ROC-AUC, Sensitivity, and Specificity.

Statistical analysis

Quantitative indicators were presented as median (Me) and interquartile range [ Q25-Q75], maximum and minimum values; qualitative variables were presented as absolute and relative frequencies. Ratios of qualitative features are presented as fractions (%). Comparison of two independent groups for quantitative data was performed using Student’s t-test for indicators conforming to the law of normal distribution and using Mann-Whitney test for indicators not conforming thereto. Qualitative variables were compared with each other using the chi-square test (χ2) and double-tailed Fisher’s exact test.

Data processing was performed using the IBM SPSS Statistics 23 software package (SPSS. Inc., Chicago, IL, USA).

R 4.2.3. software package was used to build and develop predictive models and to assess their performance.

Ethical expertise

The study protocol was approved by Endocrinology Research Centre’s local ethical committee on 20 February 2013, session record no. 2. Extract no. 12 from the record was made on 29 June 2022. All patients included in the study have given their informed consent to participate therein.

RESULTS

Subjects of the study

The study included 223 patients (163 females, 60 males) with ACTH-dependent endogenous hypercortisolism, of which 175 had Cushing disease and 48 had EAS. General characteristics of patients, main laboratory-determined parameters and results of comparative analysis are presented in Table 1. The patients were divided into two groups according to the final clinical diagnosis.

At the first stage of the study, we analysed possible differences in clinical-demographic and laboratory-instrumental parameters between the comparison groups (Table 1). Patients in both groups were predominantly female; the groups did not differ in age at disease onset, body mass index or the results of low-dose dexamethasone test (LDT). EAS patients had higher levels of all major hormonal parameters (p<0.001) and lower potassium values (p<0.001) in the active stage of the disease. Negative results of highdose dexamethasone test and no visible pituitary adenoma in brain MRI occurred statistically significantly more frequently in the EAS group of patients vs. Cushing disease patients (Table 1). Out of the 223 patients, three patients did not undergo MRI imaging of the pituitary gland because of a pacemaker, claustrophobia, and morbid obesity (BMI at 57.5 kg/m²).

Machine learning model for predicting the probability of EAS in patients with ACTH-dependent endogenous hypercortisolism

Initially, the total group of patients (n=223) was randomly divided into two sets at 80-20 ratio: the training set (n=179) and the test set (n=44) (Figure 1).

As a result of data pre-processing and selection of the most informative markers using random forest method and Boruta algorithm, the following were selected as final variables for EAS classification and prediction: morning ACTH level, potassium level, 24-h urinary cortisol, late-night blood cortisol, late-night salivary cortisol, and the largest size of pituitary adenoma on brain MRI. At the end of the selection procedure, each marker was assigned a value expressing its informativeness (Figure 2). The higher the value, the more valuable the marker is for prediction and classification. As a result, we obtained a list of markers that can be ranked according to their relevance to the classification and prediction (Figure 2).

Variables that turned out to be less relevant were removed from the analysis: age at disease onset, sex, BMI, LDT result, and pituitary adenoma visibility on brain MRI (Figure 2).

Table 2 presents performance parameters of predictive models. These parameters were obtained through machine learning algorithms applied to the training sample (n=179). Comparative analysis of predictive accuracy indicators enabled us to identify certain differences between the models. Thus, the quality of prediction by AUC, sensitivity and specificity metrics for the model based on the binary decision tree (CART) was 0.694; 87.3 and 48.7%, respectively, which indicates that the accuracy of the model when tested on the analysed cohort of patients is insufficient and needs to be improved. In turn, the GLMNET model provided good sensitivity (96.4%) and AUC (0.840) on the training sample yet with the lowest specificity (28.2%). The highest ROC-AUC values were obtained for the Naïve Bayes and the Gradient Boosted Modelling models (AUC of 0.856 and 0.867, respectively), which means they have a very good predictive accuracy.

The Generalised Boosted Modelling (GBM) demonstrated the best predictive ability of all trained machine learning models in all three final metrics (ROC-AUC, sensitivity, and specificity).

The final model was trained with the following hyperparameters (Table 3).

The results of testing the accuracy of classification performed by the Generalised Boosted Modelling (GBM) on the test sample (n=44) are presented in Figures 3 and 4 and Table 4. The accuracy of our model of differential diagnosis is as follows: probability of correct EAS prediction (predictive value of a positive result) is 87.5%; probability of correct EAS exclusion (predictive value of a negative result) is 94,4%. This method of differential diagnosis using machine learning algorithm enables EAS predicting with a sensitivity of 77.8% and a specificity of 97.1%.

The significance of clinical markers determining prediction accuracy of the Generalised Boosted Modelling (GBM) model is shown in Figure 5. Six factors were found to have a dominant influence on the resulting variable, with the most significant parameters being early morning ACTH level, minimum potassium value during the active stage of the disease, 24-h urinary cortisol and late-night blood cortisol. Late-night salivary cortisol was less influential, and the contribution of the maximum size of pituitary adenoma at brain MRI scan was minimal (Figure 5).

Examples of using the developed machine learning model for differential diagnosis of ACTH-dependent endogenous hypercortisolism

Clinical example 1

Patient S., male, 38, was admitted to Neuroendocrinology and Osteopathy Department of Russia’s Endocrinology Research Centre. Results of Patient S. examination in the hospital are presented in Table 5. Brain MRI with intravenous contrasting revealed diffuse heterogeneity of the anterior pituitary structure; no convincing data for pituitary microadenoma were obtained.

Our predictive model determined the probability of EAS in this patient at 11%; i.e., in this case the model enabled us to conclude in favour of Cushing disease in Patient S.

To confirm this conclusion using our machine learning model we compared the results of histological and immunohistochemical examinations of Patient S’s pituitary tumour tissue that had been removed. Histology findings: pituitary adenoma from basophilic cells. Immunohistochemical findings: corticotropinoma (positive ACTH expression). Thus, the conclusion made on the basis of our non-invasive method of differential diagnosis based on machine learning algorithm coincided with the results of pathomorphological examination.

Clinical example 2

Patient O., female, 32, was admitted to Neuroendocrinology and Osteopathy Department of Russia’s Endocrinology Research Centre. Results of Patient O. examination in the hospital are presented in Table 6. Brain MRI with intravenous contrasting revealed a 5.5×4 mm pituitary adenoma.

To confirm this conclusion, we compared the results of histological and immunohistochemical examinations of the removed lung tumour tissue. Histological findings: typical lung carcinoid. Immunohistochemical conclusion: focal expression of ACTH in the tumour cells. Thus, our machine learning model correctly classified the type of ACTH-dependent hypercortisolism in this patient, which was confirmed by the results of pathomorphological examination of the removed lung tumour tissue.

DISCUSSION

In this study, we developed Russia’s first machine learning-based model for and proposed a method of non-invasive differential diagnosis of ACTH-dependent forms of endogenous hypercortisolism (Cushing disease and EAS) which enable prediction of EAS probability in a patient with high accuracy (93.2%), sensitivity (77.8%) and specificity (97.1%). The use of our algorithm can significantly reduce the need for invasive differential diagnosis through bilateral selective blood sampling from inferior stony sinuses.

Predictive research in clinical medicine is one of the most promising and rapidly developing areas. Publications in international literature indicate growing interest for the development of predictive models for non-invasive verification of the etiology of endogenous hypercortisolism on the basis of modern machine learning technologies. Currently, active research is being undertaken to improve and increase the models predictive value. For example, machine learning models can effectively predict both early postoperative outcomes in patients with pituitary adenoma, including patients with Cushing disease [11], and long-term remission after removal of pituitary corticotropinoma [12]. However, the smallest amount of research in this area is devoted to differential diagnosis of various forms of hypercortisolism.

Isci S. et al. [10] were the first to propose a method of differential diagnosis of various forms of endogenous hypercortisolism (Cushing disease, Cushing syndrome, subclinical hypercortisolism) using machine learning methods based on data from 241 patients and including 11 predictor variables. Researchers applied various machine learning methods to create predictive models. Their results showed that the Random Forest (RF) algorithm significantly outperformed predictive abilities of other machine learning algorithms. Application of the model enabled classification of hypercortisolism syndrome with an average accuracy of 92%. A binary classification algorithm based on RF ‘one vs all’ demonstrated a sensitivity of 97.6%, accuracy of 91.1%, and specificity of 87.1% in confirming or excluding endogenous hypercortisolism in the test sample. Multiclass classification with RF showed an average sensitivity by class at 91.8%, average specificity at 97.1%, and average accuracy at 92.1% when classifying different forms of endogenous hypercortisolism in the test dataset. However, the algorithm developed by our international colleagues cannot provide differential diagnosis of Cushing disease and EAS the as the selected cohort had no patients with EAS.

The first research into the capacity of machine learning methods in the area of differential diagnosis of ACTH-dependent forms of endogenous hypercortisolism (Cushing disease and EAS) was conducted by Chinese scientists Lyu X. et al. [13][14]. Data from 311 patients, including 47 patients with EAS, were used for modelling. Based on multivariate logistic regression analysis, out of 11 variables, female sex (odds ratio (OR) 3.030), hypokalaemia (OR 0.209), ACTH level (OR 0.988), visibility of pituitary adenoma on MRI (OR 8.671), and positive HDT (OR 2.768) were identified as predictors associated with Cushing disease in patients with ACTH-dependent endogenous hypercortisolism [13]. Out of the eight models created, GBM yielded the highest values of the area under the ROC curve (AUC 0.980±0.02), but the RF model was the best for predicting Cushing disease among all machine learning models. RF algorithm sensitivity was 98.9%; its specificity was 87.9%, and its AUC was 0.976. Application of the predictive model to the training sample demonstrated high sensitivity (98.4%) and specificity (100%) of the algorithm, but in the test dataset, the sensitivity and specificity values were 95% and 71.4%, respectively. According to the results, the three most significant variables were serum potassium level, size of pituitary adenoma on MRI and ACTH values. The RF algorithm-based model had the largest area under the ROC curve (AUC 0.984 (95% CI 0.950-0.993)) compared to the AUC for LDT and HDT (p<0.001), but no statistically significant differences were found when comparing the areas of the selected model and the ‘gold standard’ method, i.e., bilateral selective blood sampling from inferior stony sinuses using a stimulation agent [14].

In contrast to the aforementioned study, in addition to early morning ACTH levels and potassium levels, we included 24-h urinary free cortisol, late-night blood cortisol, late-night salivary cortisol, and the largest size of pituitary adenoma on brain MRI into the set of universal predictors providing high predictive accuracy. EAS is believed to have a more severe course with a rapid increase in the severity of clinical symptoms and higher levels of all major hormonal parameters [15]. Decrease in blood potassium occurs in hypercortisolism of any etiology and is not a specific marker of EAS despite the fact that the frequency of hypokalaemia in ectopic ACTH production syndrome can reach 82.6% against 21% of cases in Cushing disease (p=0.001), according to the results of a comparative study by Attr B. et al. [15]. According to our data, the results of preoperative laboratory examination were statistically significantly different between patients with Cushing disease and EAS (p<0.001), which is consistent with the aforementioned study.

In our study, prediction of EAS probability was based on the analyses of simple laboratory and instrumental parameters available in routine clinical practice. The high quality of predictor variables in our study was ensured by a multistage selection procedure, which included assessment of their informativeness using automatic methods, in particular the Boruta algorithm, as well as selection of clinical features that ensure prediction accuracy. The use of this algorithm enabled ranking individual predictors in terms of their contribution to the predictive potential and determining the variables required for a high-performance model.

The results of our study are consistent with those achieved by international scholars who also find that more sophisticated machine learning methods surpass the capabilities of traditional statistical models, including models based on logistic regression. However, in contrast to previous research, in our study the best quality metrics in predicting EAS probability were observed when using a machine learning method based on gradient boosting (Generalised Boosted Modelling, GBM); therefore, this model seems preferable for use in clinical practice.

Clinical significance of the results

The model for EAS differential diagnosis and probability prediction developed in this study can be used as a piece of software (a calculator) and provides a decision support tool. The calculator may include fields where parameter values are entered (results of laboratory tests: early morning ACTH, late-night blood cortisol, late-night salivary cortisol, 24-h urinary free cortisol, potassium, and the largest size of pituitary adenoma on brain MRI). Based on the model judgement, the physician will be able to personalise the patient management algorithm, select the optimal set of therapeutic measures and, in the event the model yields a positive prediction, select additional diagnosis activities which will ultimately improve the treatment effectiveness, reduce the sick leave duration, eliminate unjustified surgical interventions and reduce the patient disability rate.

Limitations of this study

Limitations of this study include a relatively small sample size, which is largely due to the orphan character of the underlying disease (Cushing disease and EAS). A single-centre study design is also a limitation which may reduce the predictive value of the model in external validation.

Areas of further research

Areas of further research could include the expansion of the training sample by involving data from other medical institutions of the Russian Federation, which will contribute to improving the model accuracy.

CONCLUSION

A machine learning predictive model enables differentiating patients with EAS and Cushing disease based on baseline clinical parameters and can be used as a primary screening of patients with ACTH-dependent endogenous hypercortisolism. The results obtained justify the use of machine learning methods in the area of medical prognosis, mark a step towards the development of personalised treatment methods and can serve as a basis for the implementation of medical decision support systems in patients with ACTH-dependent hypercortisolism.

ADDITIONAL INFORMATION

Source of funding. This study was funded by the Russian Science Foundation (Grant 9-15-00398-П).

Conflict of interest. The authors hereby declare they have had no apparent or potential conflict of interest related to the content of this publication.