Abstract
Background/Aim: We aimed to develop a novel recurrence prediction model for stage II-III colon cancer using simple auto-artificial intelligence (AI) with improved accuracy compared to conventional statistical models. Patients and Methods: A total of 787 patients who had undergone curative surgery for stage II-III colon cancer between 2000 and 2018 were included. Binomial logistic regression analysis was used to calculate the effect of variables on recurrence. The auto-AI software ‘Prediction One’ (Sony Network Communications Inc.) was used to predict recurrence with the same dataset used for the conventional statical model. Predictive accuracy was assessed by the area under the receiver operating characteristic curve (AUC). Results: The AUC of the multivariate model was 0.719 (95%CI=0.655-0.784), whereas that of the AI model was 0.815, showing a significant improvement. Conclusion: This auto-AI prediction model demonstrates improved accuracy compared to the conventional model. It could be constructed by clinical surgeons who are not familiar with AI.
Colorectal cancer is a common cancer worldwide and the second leading cause of cancer death especially in Japan. The treatment for colon cancer is established, however, recurrence after curative surgery occurs in many patients, and mortality rates from colon cancer are still high. A number of studies have reported on the development of prediction models for colon cancer recurrence using various clinicopathological factors (1-7). These models use conventional statistical analysis, such as multivariate analysis and nomogram, and have been evaluated to be very good in terms of accuracy. Yet, the reported values of the area under the receiver operating characteristic (ROC) curve (AUC) and concordance index (C-index) are all under 0.8 (1-7).
Recently, artificial intelligence (AI) has developed rapidly across the world. With the development of sequencing technologies and computational methods to facilitate big data analysis, AI, as a prognostic tool, has been developed to refine for precision and accuracy (8). Machine learning (e.g., deep learning) has been used to predict various outcomes in the medical field, and methods such as decision tree and gradient boosting tree, support vector machine, and artificial neural network have been applied in cancer research (9-11). In particular, research into machine learning as an effective method to generate predictive models, which can delineate important factors in cancer heterogeneity, response, and survival has gained attention in recent years (12). However, no studies have reported on the development of AI-based prediction models for colon cancer using clinicopathological databases, despite their potential to demonstrate improved predictive accuracy. Although it is difficult for clinical surgeons to develop algorithms of machine learning, the new machine learning software “Prediction One” (Sony Network Communications Inc., Shinagawa, Tokyo, Japan) enables us to evaluate the predictions easily. We attempted to use this software first in the world.
Recently, increased recognition of the role of systemic inflammatory response and nutrition status in cancer outcomes has been observed (13). Many immunological and nutritional markers have been reported to serve as prognostic factors for many kinds of cancer, including colon cancer. Among those, neutrophil-to-lymphocyte ratio (NLR) (14-16), lymphocyte-to-monocyte ratio (LMR) (17, 18), platelet-to-lymphocyte ratio (PLR) (19, 20), Japanese modified Glasgow prognostic score (mGPS) (21, 22), CRP-to-albumin ratio (CAR) (23-24), prognostic nutrition index (PNI), and controlling nutrition status (CONUT) (25, 26) are well-established markers of colon cancer recurrence and can be obtained easily preoperatively. Incorporating these factors can improve the predictive accuracy of models.
In this study, we aimed to develop a novel prediction model using the auto-AI software “Prediction one”, which demonstrates improved accuracy compared to conventional statistical analysis models for predicting recurrence in patients with stage II-III colon cancer.
Patients and Methods
Patients. Data from 814 patients who had undergone curative surgery for pathological stage II-III (pStage II-III) colon cancer between January 2000 and October 2018 at the Tokyo Medical University Hospital were acquired. The following 29 variables were extracted from this database: age, sex, body mass index (BMI), use of insulin, normal/emergency surgery, sidedness (tumor location), laparoscopic/open surgery, macroscopic classification, tumor size, histological type, pathological T-stage (pT-stage), pathological N-stage (pN-stage), pathological stage (pStage), number of harvested lymph nodes (NHL), number of metastatic lymph nodes (NML), lymphatic invasion (ly), venous invasion (v), adjuvant chemotherapy (adj), postoperative complication, surgical site infection, carcinoembryonic antigen (CEA), carbohydrate antigen 19-9 (CA-19-9), NLR, LMR, PLR, CAR, mGPS, PNI, and CONUT. Patients without any of these 29 variables were excluded. Finally, 787 patients were retrospectively reviewed. Generally, patients are admitted to our hospital a few days before surgery, and laboratory values are obtained on the day of admission. This study was conducted according to the guidelines of the Declaration of Helsinki. The Review Board of the Tokyo Medical University Hospital approved the study (T2019-0060), and informed consent was obtained from the patients.
Treatment. We performed curative surgeries for all patients. We generally perform postoperative systemic adjuvant chemotherapy for pStage III colon cancer at our hospital. In the present study, we performed adjuvant chemotherapy in 95 of 399 patients (23.8%) with pStage II disease and 241 of 388 (62.1%) patients with pStage III disease. We generally use oxaliplatin-based or 5-fluorouracil-based regimens. The indications of postoperative chemotherapy and regimens were selected based on patient characteristics, such as fitness and age, as well as physician discretion.
Follow-up. For surveillance, we followed up the patients with stage II-III tumors for five years after surgery. The median follow-up period was 83.2 months (range=25.3-232.7 months). Specifically, we performed tumor marker measurements initially every 3 months for up to 2 years, followed by additional computed tomography scans and tumor marker measurements every 6 months for the next 3 years. Recurrence site and postoperative period were recorded at the first recurrence. Local recurrence (LR) was defined as any histological or clinical evidence of tumor recurrence near the primary site. Recurrence at distant sites, such as the liver and lung, was defined as distant metastasis (DM).
Statistical analysis. We defined relapse-free survival (RFS) as the interval between the operation date and the recurrence date, or death from underlying disease. We censored observations when patients died from non-colon cancer-related causes. ROC analysis was used to analyze the prognostic and predictive accuracy. We analyzed survival characteristics using the Kaplan-Meier method.
Univariate analysis was performed using the chi-square test or the Kruskal-Wallis test. Multivariate analysis was performed using binomial logistic regression to calculate the effect of variables on recurrence after curative surgery. The measure of the effect of each variable on recurrence is presented as an odds ratio (OR) and used to identify independent risk factors. Predictive accuracy was assessed by the AUC in ROC analysis.
All statistical tests were performed by SPSS software (IBM® SPSS® Statistics for Windows, Version 25.0; IBM, Chicago, IL, USA), and p>0.05 was considered to indicate statistically significant differences.
Artificial intelligence. We used Prediction One (Sony Network Communications Inc.) machine learning software to predict recurrence of pStage II-III colon cancer using the same dataset for normal statistical analysis. The software generates feature vectors from the dataset using standard preprocessing methods such as one-hot encoding for categorical variables and normalization for numerical variables. Gradient boosting tree and a neural network are used as supervised machine learning models, each trained with hyperparameter tuning, and an ensemble model of both trained models is constructed. To evaluate the accuracy of the AI model, the AUC was calculated with internal validation. Prediction One also evaluates the “importance of variables (IOV)” using a method based on permutation feature importance (27). This method is used to calculate the difference in model output when a single variable is removed. The value of difference in the model output indicates how much the model depends on the variable. The value of difference is computed for each covariate, and then averaged over those in the dataset.
Results
Patient and tumor characteristics. Baseline patient and tumor characteristics are shown in Table I. In overall patients, 5-year overall survival and 5-year RFS rates were 93.2% and 89.8%, respectively (Figure 1A and B). Overall, 97 patients (12.3%) had recurrence, including 21 (2.7%) with LR and 76 (9.6%) with DM. The 5-year cumulative recurrence rate was 9.8% (Figure 1C).
Cut-off values of inflammation and nutrition biomarkers. Regarding RFS, we analyzed the accuracy and cut-off values of each inflammation and nutrition marker using ROC curves. AUCs for NLR, LMR, PLR, PNI, CAR, CONUT, and mGPS were 0.516, 0.526, 0.506, 0.514, 0.504, 0.521, and 0.489, respectively, with cut-off values of 2.1, 5.0, 0.74, 0.1, 47.8, and 1.0, respectively (the cut-off value for mGPS was not calculated, as the AUC was under 0.5).
Development of a prediction model using binomial logistic regression analysis. Table II shows the results of univariate and multivariate analyses for recurrence. All 29 covariates were included in the model. In the multivariate analysis, BMI [p=0.03, OR=4.448 (95%CI=1.156-17.122)], number of metastatic lymph nodes [p=0.006, OR=1.316 (95%CI=1.080-1.604)], CEA [p=0.003, OR=11.0 (95%CI=2.245-53.685)], CA19-9 [p=0.01, OR=8.293 (95%CI=1.669-41.215)], and differentiation [p=0.013, OR=104.454 (95%CI=2.628-4151.47)] were found to be independent risk factors for postoperative recurrence. The ROC curve of this model for predicting postoperative recurrence is shown in Figure 2. The AUC was 0.719 (95%CI=0.655-0.784).
Artificial intelligence analysis. We used Prediction One to analyze the model including the same covariates to predict postoperative recurrence of stage II-III colon cancer. Data from 687 and 100 patients were used for the learning model and validation model, respectively. The ROC curve of the AI model is shown in Figure 3. The AUC was 0.815, suggesting a significant improvement compared to the multivariate model. Prediction One was also used to calculate the IOV of each factor for recurrence. Factors with IOV ≥0.020, in decreasing order, were CEA (0.046), NML (0.036), adjuvant chemotherapy (0.030), T-stage (0.029), BMI (0.024), and CA19-9 (0.023).
Discussion
The strength of this study is that a clinical surgeon who is not familiar with AI could construct a highly accurate prediction model using simple auto-AI. There are many established strategies for colon cancer treatment. However, LR and DM occur in many patients even after curative operation. This study aimed to develop a novel AI model for predicting colon cancer recurrence with various clinicopathological factors using simple auto-AI.
With regard to the accuracy of previously reported models from conventional statistical analysis, reported values of the AUC or C-index are all under 0.8 (range=0.550-0.78) (1-7). In the present study, while the accuracy of the conventional statistical model (AUC, 0.714) was comparable to those reported by others, the accuracy of the AI model (AUC, 0.815) was significantly higher. These results suggest the usefulness of the AI model for predicting recurrence in colon cancer patients. Moreover, this model, which consists of simple clinicopathological factors that are easy to obtain, can be constructed at low cost. It should be emphasized that ‘Prediction One’ is easy for a clinical surgeon to use for artificial intelligence analysis.
In Japan, colon cancer is diagnosed and treated according to oncological staging by the TNM classification. However, discrepancies exist between staging and clinical practice regarding both prognosis and choice of treatment. The 2019 Japanese Society for Cancer of the Colon and Rectum Guidelines for the treatment of colon cancer thus recommend that additional clinicopathologic factors should be considered for staging, although specific criteria have yet to be established (27). The identification of a true prognostic factor for use in clinical practice is urgently required. Currently, no evidence exists regarding prognostic factors for pStage II and III colon cancer that are useful in the selection of postoperative treatment. Therefore, our model using AI can offer an excellent option to tackle these issues.
The use of machine learning, which includes supervised or unsupervised approaches, has increased in the field of clinical research. In the supervised approach, the expected output quality is known. Deep neural networks are currently the foundation for many modern AI applications (28) due to their prediction performance and flexibility in application to various types of problems. Binomial logistic regression is a (generalized) linear model, whereas Prediction One uses a non-linear model (gradient boosting tree and a neural network) that can handle combinations of features. This may be the reason for the observed difference in accuracy.
While AI appears to be powerful in terms of outcome and prediction, it also suffers from opacity. That is, it is difficult to understand the internal mechanism of analysis (“black-box problem”) (29), which is problematic because entrusting important decisions to a system that is difficult to explain itself presents obvious dangers (30). In the present study, on the contrast, Prediction One can calculate not only the accuracy of the model, but also the contribution of each factor to the result. It enabled us to get an understanding of the model. The independent risk factors identified in the multivariate analysis and those found to have high contribution in the AI analysis were similar in some aspects. For example, tumor markers and BMI were identified by both analyses. However, NML, T-stage, and adjuvant chemotherapy were only identified in the AI analysis.
There are some limitations in the present study. First, we used a single-center retrospective design. Second, there are no records of comorbidities such as hematologic or autoimmune disease. These diseases may have influenced inflammation and nutrition values. Third, some covariates were not included in this model, such as immune parameters and molecular parameters, because there was no information in some patients. Fourth, although Prediction One partly resolved the “black-box problem” by calculating IOV, we could not completely exclude this problem from the present analyses. Fifth, external validation was not performed. Finally, the number of patients is small for deep learning analysis. As this is a preliminary study, prospective studies with a large number of patients and external validation will be necessary to further improve the AI model.
In conclusion, we developed a novel AI prediction model with improved accuracy compared to the conventional statistical analysis model for predicting recurrence in patients with stage II-III colon cancer in this preliminary study. This model can be used as an alternative to conventional models for determining treatment strategies after curative surgery.
Footnotes
Authors’ Contributions
J.M. wrote the main manuscript text and Y.O., R.U., T.T., K.K., H.K., M.E. and T.I. acquired data for the work. All Authors reviewed the final manuscript. KK, YN and AT drafted the work or revised it critically for important intellectual content.
Conflicts of Interest
The Authors declare that there are no conflicts of interest in relation to this study.
- Received July 9, 2021.
- Revision received July 29, 2021.
- Accepted August 3, 2021.
- Copyright © 2021 International Institute of Anticancer Research (Dr. George J. Delinasios), All rights reserved.