Abstract
Background/Aim: The aim of this study was to analyze the survival predictions obtained from a web platform allowing for computation of the so-called Bone Metastases Ensemble Trees for Survival (BMETS). This prediction model is based on a machine learning approach and considers 27 prognostic covariates. Patients and Methods: This was a retrospective single-institution analysis of 326 patients, managed with palliative radiotherapy for bone metastases. Deviations between model-predicted survival and observed survival were assessed. Results: The median actuarial survival was 7.5 months. In total, 59% of patients survived for a period shorter than predicted. Twenty percent of the predictions of the median survival deviated from the observed survival by at least 6 months. Regarding actual survival <3 months (99 of 326 patients), the BMETS-predicted median survival was <3 months, i.e. correct in 67 of 99 cases (68%), whereas the model predicted a median of 4-6 months in 16 (16%) and of >6 months in another 16 cases. Conclusion: The model predicted survival with high accuracy in a large number of patients. Nevertheless, if the model predicts a low likelihood of 3-month survival, actual survival may be very poor (often 1 month or less). Also, in patients who died within 3 months from the start of radiotherapy, the model often predicted longer survival (16% had >6 months predicted median survival). It would, therefore, be interesting to feed the U.S. database utilized to develop the BMETS with additional poor-prognosis patients to optimize the predictions.
A large number of cancer patients worldwide receive palliative radiotherapy for bone metastases (1). However, the treatment scenarios are highly variable and include pain-alleviating irradiation of uncomplicated bone metastases, post-operative radiotherapy after surgical stabilization, irradiation of impending or established spinal cord and/or nerve root compression and others (2, 3). The survival prognosis of these patients is heterogeneous, too. Pain might be present already as the first sign of cancer in patients expected to survive for several years, or in the pre-terminal and terminal phase when all systemic therapies have been stopped due to futility and/or the patients’ reduced general condition (4-8). This heterogeneity causes uncertainty regarding treatment decisions, such as the decision to irradiate at all and the choice of fractionation regimen (9-11). Several groups have proposed prognostic models, which may support decision making (12-14). Recently, new technology has resulted in improved opportunities to add complexity to such models, e.g. by integrating a much larger number of prognostic factors than older methods were able to handle. Alcorn et al. have utilized a machine learning approach to analyze whether they could optimize survival estimation for patients with symptomatic bone metastases (15). Their so-called Bone Metastases Ensemble Trees for Survival (BMETS) predict survival using 27 prognostic covariates. As briefly discussed in a recent correspondence to this study (16), some of the 27 covariates are prone to practice variation, in particular on an international level, i.e. between different healthcare systems. Examples include access to in-patient care and the type of systemic therapy. Thus, our group performed an independent validation study in a different geographical region (Norway as compared to USA), which has a different healthcare system (publicly-funded). Due to space limitations for correspondence, only a shortened report of the resulting concerns has been published. The present paper provides a broader set of results.
Patients and Methods
Analogous to a previous validation approach (14), our single-institution database that includes unselected patients irradiated for complicated or uncomplicated bone metastases from histologically verified primary tumors (both completed and interrupted treatment courses according to the intention-to-treat principle, 2009-2018) was analyzed. Radiotherapy prescription was individualized (often 3 Gy ×10, 4 Gy ×5 or 8 Gy ×1), as was systemic therapy. Staging consisted of computed tomography. If clinically relevant, other modalities were added to clarify computed tomographic findings, e.g. isotope bone scan, ultrasound, and positron-emission tomography. Routine blood tests were assessed during treatment planning approximately 1 week before radiotherapy.
Alcorn et al. (15) developed a web platform for data entry and display of BMETS-predicted survival probabilities (https://oncospace.radonc.jhmi.edu/Tools/PalliationPrediction.aspx), which was utilized in the present study. The model-based Kaplan-Meier curves are truncated at 12-month follow-up and therefore, median survival cannot be assessed if the predicted survival is longer than 12 months. Both 3- and 12-month survival probabilities were tabulated [Results reported in (16)]. Furthermore, the predicted median survival was recorded.
Overall survival (time to death) from the first day of radiotherapy was calculated employing the Kaplan–Meier method (SPSS 25; IBM Corp., Armonk, NY, USA). The minimum follow-up was 6 months. The median follow-up of 26 censored patients was 48 months. As mentioned above, this database created for the purpose of quality-of-care analyses has already been utilized and does not require additional approval by the local Ethics Committee (REK Nord).
Results
The aforementioned web platform was utilized to analyze the BMETS performance in 326 patients (Table I). Their median age was 67 years (minimum 32 years, maximum 90 years). The three most common cancer types were located in the prostate, breast or lung. Most patients who received systemic therapy had endocrine treatment (40%) or chemotherapy (30%). Treatment to more than one target volume, e.g. spine and femur, was common (59%). Eight patients (2%) failed to complete their prescribed course of radiotherapy. The median actuarial survival was 7.5 months.
Baseline data.
The median survival according to the Kaplan-Meier curve derived from the web platform was ≤3 months in 58 patients (18%). Their median survival was 35 days (range=5-245) and 7 survived for >3 months. In 94 patients (29%), a median survival of more than 12 months was predicted. Of these, 20 died after less than 12 months (minimum 81 days). In total, 59% of all 326 patients survived shorter than predicted. Forty-five predictions of the median survival time in the 232 eligible patients (those whose Kaplan-Meier curves were not truncated at 12 months, see Methods) were within 1 month of the actual survival time (19%). Another 53 (23%) were within 2 months and 37 (16%) within 3 months. However, 47 (20%) deviated by at least 6 months. Figure 1 shows the observed deviations from predicted survival in these 232 patients (mean=60.5 days).
Deviation from predicted median survival in days.
Regarding actual survival <3 months (99 of 326 patients), the BMETS-predicted median survival was correct in 67 of 99 cases (68%), whereas the model predicted a median of 4-6 months in 16 (16%) and of >6 months in another 16 cases.
Discussion
The original BMETS study (15) included 397 patients (treatment period January 2007 to January 2013). Two previously validated, simpler models were also studied: Chow’s 3-item and Westhoff’s 2-item tools (12, 17). The model performance was assessed using cross-validation procedures and measured by time-dependent area under the curve (tAUC) for all 3 models. For temporal validation, a separate data set comprised of 85 patients treated in 2018 at the same U.S. institution was used. Median survival was 6.4 months (comparable to the present data, 7.5 months). BMETS (27 prognostic covariates, such as age, gender, primary cancer site, performance status, steroid and opioid medication, weight loss, pattern of other metastases) outperformed the simpler models at each time. For the temporal validation set, respective tAUC was 0.86, 0.82, and 0.78. These statistically sound validation methods lack an intuitively understandable dimension, telling clinicians how much agreement or disagreement they can expect if they decide to use the web platform in clinical routine. In a previous nomogram study, our group developed a visual validation plot comparable to the one shown in Figure 1 (18). This method illustrates how much deviation from the predicted median survival can be observed in individual patients. Even a good and complex prognostic model such as the BMETS is associated with large deviations in a minority of patients.
Prediction of very short survival was not the main goal of the BMETS study. Nevertheless, the present results and those published in (16) suggest that, if the model predicts a low likelihood of 3-month survival, actual survival may be very poor. This means that some patients are unlikely to experience the potential benefit palliative radiotherapy may cause if survival is long enough. Also, in patients who died within 3 months from the start of radiotherapy, the model often predicted longer survival (16% had >6 months predicted median survival). It would, therefore, be interesting to feed the U.S. database with additional poor-prognosis patients to optimize the predictions. A possible explanation for deviating results are international practice variations such as the threshold for hospitalization and the choice of systemic therapy, variables that are part of the prediction model. For example, a 70-year old male with prostate cancer, bone-only metastases, Karnofsky performance status of 70, not admitted to hospital and on endocrine systemic therapy has a predicted 12-month survival probability of 56%. This probability drops to 48% if the same patient is admitted to hospital, and to 31% if the systemic therapy changes to intravenous chemotherapy and the patient is still hospitalized (calculated at https://oncospace.radonc.jhmi.edu/Tools/PalliationPrediction.aspx on December 31, 2020).
The BMETS approach has several strengths such as the large number of prognostic covariates, careful validation, comparison to simpler models and easy-to-use web platform. However, weaknesses of this model need to be acknowledged, e.g. the impact of practice variations on survival predictions and the time needed to collect and enter all covariates. A larger multi-institutional or even international database might be a good starting point for refinement of the model, aiming at further improvement of the prediction accuracy.
Footnotes
Authors’ Contributions
CN participated in the design of the study and performed the statistical analysis. CN, BM and RY conceived the study and drafted the article. All Authors read and approved the final article.
Conflicts of Interest
The Authors declare that they have no conflicts of interest.
- Received January 11, 2021.
- Revision received January 29, 2021.
- Accepted February 11, 2021.
- Copyright © 2021 International Institute of Anticancer Research (Dr. George J. Delinasios), All rights reserved.
References
In this issue
Jump to section
Related Articles
Cited By...
- Radiotherapy for Bone Metastases in Patients With Excellent Performance Status: Patterns of Care and Prognostic Factors for Survival
- Short Communication: Results of a Consensus Conference on Radiotherapy for Brain and Bone Metastases Within the Interreg-Project TreaT
- Independent Validation of a Risk Stratification Model Predicting Survival in Elderly Patients Irradiated for Bone Metastases
- Current Radiotherapy Concepts Regarding Brain and Bone Metastases in Centers Participating in the German-Danish Interreg-Project TreaT
- Independent Validation of a Risk Stratification Model Predicting Survival in Patients With Metastatic Hormone-sensitive Prostate Cancer
- Independent External Validation of the METSSS Model Predicting Survival After Palliative Radiotherapy