Research ArticleClinical Studies

Simultaneous Truth and Performance Level Estimation Method for Evaluation of Target Contouring in Radiosurgery

HELENA SANDSTRÖM, IULIANA TOMA-DASU, CAROLINE CHUNG, JONAS GÅRDING, HIDEFUMI JOKURA and ALEXANDRU DASU
Anticancer Research January 2021, 41 (1) 279-288; DOI: https://doi.org/10.21873/anticanres.14774

Abstract

Background/Aim: The problem of lack of standardisation in target delineation and herewith the variability of target contours in Gamma Knife radiosurgery is as severe as in linac-based radiotherapy in general. The first aim of this study was to quantify the contouring variability for a group of five radiosurgery targets and estimate their true-volume based on multiple delineations using the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm. The second aim was to assess the robustness of the STAPLE method for the assessment of the true-volume, with respect to the number of contours available as input. Patients and Methods: A multicentre analysis of the variability in contouring of five cases was performed. Twelve contours were provided for each case by experienced planners for Gamma Knife. To assess the robustness of the STAPLE method with respect to the number of contours used as input, sets of contours were randomly selected in the analysis. Results: A high similarity was observed between the STAPLE generated true-volume and the 50%-agreement volume when all 12 available contours were used as input (90-100%). Lower similarity was observed with smaller sets of contours (10-70%). Conclusion: If a high number of input contours is available, the STAPLE method provides a valuable tool in the estimation of the true volume of a target based on multiple contours as well as the sensitivity and specificity for each input contour relative to the true volume of that structure. The robustness of the STAPLE method for rendering the true target volume depends on the number of contours provided as input and their variability with respect to shape, size and position.

Key Words:

The success with respect to treatment outcome and normal tissue toxicity of stereotactic radiosurgery, delivered in one large fraction of radiation and without margins, relies on the accuracy in contouring the target and the organs-at-risk (OAR). However, despite its key role in the success of treatment, the delineation of the target and OARs in radiosurgery appears to vary considerably between practitioners and this could have unwanted clinical implications (1-3). This state of the affairs points towards the need of harmonising the clinical practice with respect to contouring the various structures in radiosurgery through standardisation. In order to issue guidelines for target and OAR delineation, leading to standardisation, one has to establish clinical standards with respect to delineation, which is not a trivial task. A standard, by definition, represents the set of accepted criteria against which the quality of the delineation can be assessed. Thus, in order to establish a standard for the definition of the target volume in radiosurgery, for example, one would have to know the true target volume. The true value of a quantity in general is obtained by a perfect measurement and therefore is by nature indeterminate. If several contours are available for one given target, as resulting from the delineation based on several imaging techniques by one observer, or from the delineation performed by several observers based on one single image, or a combination of the two, i.e. delineations performed by several observers based on different morphological and functional images, one could determine the variability in delineating the structure by comparing the individual targets with respect to size, shape and position and determine the most probable true volume by applying statistical methods. Several approaches could be employed to determine the true volume of a given structure, in general, or of a given radiosurgery target, in particular. Among them one could mention the Simultaneous Truth and Performance Level Estimation (STAPLE) algorithm (4), an expectation-maximization algorithm used for the probabilistic determination of the true volume of a structure based on several input volumes generated by trained experts. In addition to rendering a probabilistic estimation of the” true segmentation”, the STAPLE method also allows the calculation of the sensitivity and specificity of each segmentation used as input. The STAPLE method was previously used in studies of variability in normal structure and target delineation in conventional radiotherapy (4-8) but, to the best of our knowledge, it was not extensively applied to radiosurgery targets except for few studies in which STAPLE was used for target volume definition as part of projects aiming at recommending consortium consensus guidelines (9, 10). However, to the best of our knowledge, there is no other study in which the systematic analysis of the dependence of the outcome of the analysis based on the STAPLE method on the number of the input contours was performed for radiosurgery targets. Thus, the aim of this study was to assess the feasibility of applying the STAPLE method for determining the true volume of several common radiosurgery targets based on input volumes delineated by experienced practitioners of radiosurgery, determine the sensitivity and specificity of each of the contours used as input and assess the robustness of the method with respect to the number and degree of variability of the input structures. Furthermore, the true volumes generated by STAPLE are compared to the results of a so-called binary approach method previously developed by Sandström and co-workers and applied for the analysis of the variability of target and OAR delineation (1, 3). Thus, the previously introduced metric, the average volume equal to the 50% agreement volume (AV50) assumed to give an indication of the true volume using the binary approach is compared to the STAPLE-determined true volume.

Patients and Methods

Clinical cases and data collection. The patient data used in this study include five targets regarded as common for radiosurgery performed with Leksell Gamma Knife® (LGK, Elekta Instrument AB, Stockholm, Sweden), namely two cases of metastases (diameters 30 mm and 13 mm), a cavernous sinus meningioma, a pituitary adenoma and a vestibular schwannoma case. A detailed description of the patient data, including the clinical information and the complete list of images that were available for them as well as some representative examples are given in the Supplementary Material of Sandström et al. (3). For all the cases, axial T1-weighted MR images and axial contrast-enhanced T1-weighted MR images were available in addition to specific other target images.

Twelve Centres using Leksell Gamma Knife® participated in the target delineation study. They all received anonymised co-registered sets of all images available for the five cases and provided contoured targets and OARs according to their local clinical practice, as well as information regarding the participants including profession of planner, experience and planning methodology. More details regarding the design of the contouring variability study are given in Sandström et al. (2018) (3). Examples of selected images of the two metastases and the corresponding contours drawn by the participants in the study overlapped on one image slice are shown in Figure 1. Analogues images for the other three targets considered in this study, the cavernous sinus meningioma, the pituitary adenoma and the vestibular schwannoma are shown in Figure 2.

Figure 1.
Figure 1.

Examples of images showing two metastases (upper panels) and the corresponding slices together with the contours delineated by the participants in the study (lower panels). A particular colour was consistently used for contours delineated by the same particular participant in the study.

Figure 2.
Figure 2.

Examples of images showing the cavernous sinus meningioma, the pituitary adenoma and the vestibular schwannoma cases (upper panels) and the corresponding slices together with the contours delineated by the participants in the study (lower panels). A particular colour was consistently used for contours delineated by the same particular participant in the study.

Data analysis. The true volume of the five targets included in this study determined by using the STAPLE method was calculated starting from the contours provided by the twelve Leksell Gamma Knife® users participating in the study. The target segmentation inputs were converted to a binary format as described by Sandström et al. (1, 3). Starting from an initial guess (determined as the median value of all contours) of the true volume/segmentation the following steps were performed in an iterative manner until the convergence was reached: the comparison of the initial guess of the true volume with each segmentation input determining at the voxel level the probability that the voxel is correctly included in delineations by all observers (i.e. “sensitivity” of the voxel) and the probability that the voxel is correctly excluded by all the participants (i.e. “specificity” of the voxel), the calculation of a weight variable giving the probability that the true volume at the specific voxel is 1 and the re-evaluation of the performance of the input target in terms of sensitivity and specificity as described in the original paper introducing STAPLE (4).

The true volume was determined in this manner for each of the five targets considered in the study using as input all 12 individual volumes/structures/contours delineated by the participants for each of the targets.

In order to assess the robustness of the STAPLE method with respect to the number of contours used as input, the method was applied to a reduced number of input contours and the resulting volumes were compared with the true volume determined from all the contours. Thus, several sets of 6 to 11 contours randomly selected from the total of 12 available for each case were considered, thus investigating the impact of removing between 1 and 6 randomly chosen contours. In addition to the true volume calculated using the maximum number of input volumes of 12 for each case, the true volume was calculated 250 times with varying number of segmentations used as input (6-11). Minimum, maximum and mean true volumes were recorded for the various simulations after the random removal of a variable number of input segmentations.

In parallel, a previously developed method for the assessment of the target contouring variability based on the agreement volumes was also applied (1, 3). For each of the targets the 50% agreement volume, AV50, the volume of the target that 50% of participants agreed upon, was calculated. More specifically, the AV50 consists of all voxels within the 12 contours available for each of the cases that were present within at least 6 or more of the contours. Considering that the calculation is based on a matrix where contours are transferred to a voxel-based binary format and added, the result is a 3-dimensional “agreement map”. This can also be referred to a 50% voting rule, similar to the majority vote. The voxel size for this calculation was 0.5×0.5×1 mm.

The true volume was compared to the 50% agreement volume, AV50, with the use of the Agreement Volume Index (AVI) calculated as the ratio of the common (intersect) to the encompassing (union) volume of the true volume and the AV50.

Results

The results of this study show high similarity between the STAPLE generated true volume and the 50% agreement volume, AV50, for all cases in the evaluation when all 12 available contours are used as input (90-100% similarity).

Graphical examples of the results for the two metastases considered in the study are shown in Figure 3. The true volume for the small metastasis was 1.6 cm3 and the AV50 1.51 cm3 showing a good spatial co-localisation as well, as it could be seen in Figure 3. The corresponding values for the large metastasis was 10.84 cm3 (STAPLE) and 11.19 cm3 (AV50).

Figure 3.
Figure 3.

Left panels: 3D surface plots of the true volume using all the segmentations available as input together with the AV50 for the two metastases included in the study. Right panels: the 3D surface plots of the true volume using all the segmentations available as input together with the encompassing (union) volume of all individual targets delineated by the participants in the study for the two metastases.

Removal of one random contour did not change the true volume substantially but random removal of up to 5 input contours might lead to a difference between the STAPLE generated true volume and the 50% agreement volume between 10-70% for all cases.

Figure 4 shows the level of agreement between the true volume (using a variable number of volumes as input) and AV50, displayed as colour map, for the two metastases included in this study. The corresponding examples of the images illustrating the level of agreement between the true volume and AV50 for the cavernous sinus meningioma, the pituitary adenoma and the vestibular schwannoma cases are shown in Figure 5. The true volume was calculated by randomly removing 1-6 volumes from the total of 12 available as input. As expected, the discordance between the true volume and AV50 increases by decreasing the number of structures used as input.

Figure 4.
Figure 4.

Examples of images illustrating the level of agreement between the true volume and AV50 for the two metastases. The true volume was calculated by randomly removing 1-6 contours from the total of 12 available as input.

Figure 5.
Figure 5.

Examples of the images illustrating the level of agreement between the true volume and AV50 for the cavernous sinus meningioma, the pituitary adenoma and the vestibular schwannoma cases. The true volume was calculated by randomly removing 1-6 contours from the total of 12 available as input.

The minimum number of input contours required for ensuring the robustness of the STAPLE method assessed by comparing the true volume rendered by it to the AV50 depends on the magnitude of the variability in contouring the target. An illustration of the way the true volume depends on the magnitude of the variability in contouring the target is shown in Figure 6 and Figure 7 taken the large and small metastasis examples. The variability in delineation for the large metastasis is high as shown in Figure 6. Consequently, the range of true volumes increased as the number of randomly removed contours increased. The inherent variability in the contour set results in a sensitivity to which of the contours are included in the calculations. Another consequence of the variability is shown in Figure 7, where the resulting AVI range also increases for the large metastasis as the number of contours removed are increased. The opposite trend is shown for the small metastasis where the inherent variability in the data set is lower.

Figure 6.
Figure 6.

The dependence of the true volume on the number of segmentations used as input for all 5 targets: the maximum, mean and minimum true volume resulting from 250 simulations in each case is given. The actual values corresponding to a given set of simulations for 1-6 contours removed are connected with lines in order to improve the visualization of the trends.

Figure 7.
Figure 7.

The dependence of the AVI on the number of segmentations used as input for all 5 targets: the maximum, mean and minimum AVI values resulting from 250 simulations in each case are given. The actual values corresponding to a given set of simulations for 1-6 contours removed are connected with lines in order to improve the visualization of the trends.

A summary of the results giving the range of contoured volumes used as input, the range of true volumes considering 6-12 input volumes and the range of AVI values indicating the level of agreement between the true volume and the AV50 for all five targets considered in this study is given in Table I. The upper limit of the AVI, close to 1 for all cases, corresponds to the case when all 12 contours available as input were used in the calculation of the true volume, indicating that applying this probabilistic model renders similar results in terms of most likely target to be true as determining the volume on which 50% of the participants agree that belongs to the target.

View this table:
Table I.

Summary of results giving the range of contoured volumes used as input, the range of true volumes and the range of Agreement Volume Index (ratio between common and encompassing volumes, AVI) values for all five targets considered in this study.

Figure 6 shows the dependence of the true volume on the number of segmentations used as input for all 5 targets. For each of the targets 1-6 contours were randomly removed at one time and the maximum, mean and minimum true volume resulting from 250 simulations in each case was plotted. In order to better visualise the trends, the actual values corresponding to a given set of simulations were connected with lines. As it could be observed for all cases, the mean true volume appears to be rather robust with respect to the number of structures used as input. The maximum or the minimum values tend to be less robust and vary when the number of structures used as input changes. Similar results are observed for the AVI, as it could be seen in Figure 7. Thus, the dependence of the maximum, mean and minimum AVI values resulting from 250 simulations in each case on the number of segmentations used as input for all 5 targets is shown. The maximum AVI is close to unit, over 90% in all cases, even in the case when only half of the available contours were used as input for determining the true volume.

As the participants in the study were asked to contour the target according to their clinical practice without specifying what images should be used for contouring among the ones available, they all used the T1-weighted contrast enhanced axial MRI for contouring except for two, one for the vestibular schwannoma case and one for the cavernous sinus case. A comprehensive report of the images used for contouring is provided in Sandström et al. (3).

Discussion

The delineation of the target is one of the most important clinical challenges in radiosurgery (11). The high conformity to the target of the dose delivered in one single fraction dictates that the target definition in radiosurgery is standardised and accurate. Our previous studies, however, show that the variability in delineation of the target in radiosurgery, but also of the OAR, is unexpectedly high (1-3). One explanation for the high variability in delineation in radiosurgery that was postulated is the fact that, in absence of guidelines and standardisation, different clinics account differently for the potential limitations of the imaging modalities to allow for the accurate determination of the extent of the target, as well as for the infiltration of the tumour cells into the normal tissue, in case of highly infiltrative tumours. The result is, thus, a collection of structures delineated by different practitioners coming from different centres for the same clinical case. As the different contours might not only reflect the ability of the system to imagine the target but also the clinical routine regarding the planning strategy, the level of experience of the person performing the delineation and generally has a large degree of subjectivity related to the empirical character of the delineation process, the “true target volume” becomes a concept not only difficult to achieve, but also to define. Therefore, in order to circumvent the problems related to the definition of the true target volume, one could consider determining it using the STAPLE method based on multiple structures provided by different observers (4). The STAPLE method has an intrinsic limitation since the volume rendered by it as the true volume depends on the number of the input contours and the number of iterations used in the algorithm. It has previously been suggested that the number of iterations generally needed to reach convergence is lower than 20 (4). However, the robustness of the method with respect to the number of input contours, to the best of our knowledge, was not previously investigated for common radiosurgery targets.

Some other methods, such as the one based on agreement volumes employed in our previous studies, could also be used to assess the true target volume (1-3). The results appear to agree well with the true volume. The advantage of STAPLE, however, is that it intrinsically gives the sensitivity and the specificity of each structure used as input relative to the true volume. This intrinsic feature of STAPLE has many practical applications and opens up new research avenues. One of the main applications could be the testing of the personnel involved in the radiosurgery planning with respect to their capacity to correctly draw the target. Thus, after collecting targets delineated by different practitioners of radiosurgery for the same clinical target, the “true volume” could be determined by applying STAPLE and the sensitivity and specificity of each target delineated by each participant, and hence the quality of the delineation performed by an individual participant, could be assessed and compared. In addition, the influence of the experience and professional speciality of the participants on the overall delineation results could therefore be assessed. The consistence in delineation of one individual participant, in other words the intra-observer analysis, could also be assessed using this method. One might hypothesize that if highly experienced users would participate to a similar study, a smaller number of contours would be needed for determining the true volume while if the participants would be less experienced a larger number of contours and hence participants would be required if one assumes one contour per participant. The experience of the participants in this study, however, was rather heterogeneous, and therefore the study is not expected to be biased from this point of view. Furthermore, the study was intentionally designed to include participants with different experience in order to overcome the issues related to the unstandardized definition of the experience. All the participants, however, are performing target delineation on current bases at their home institutions.

One other aspect that should be mentioned in this context is that the specificity of a given delineation, thus the relative extent of normal tissue volume correctly identified as not belonging to the true volume to the total volume of the tissue included in the analysis, depends on the size of this total volume. Thus, the higher the volume encompassing the delineated structure in which the analysis is performed, the higher will the specificity of a given contour appear to be. Although trivial when pointed out, this feature of the STAPLE method should be kept in mind when applying it and the volume in which the analysis of sensitivity and specificity is performed should be specified and kept constant throughout the study. Regarding the methodology employed in this study, it should also be mentioned that the choice of the median value of the contours as the initial guess was based on a preliminary analysis in which several other initial guesses such as the common volume (intersect) and the encompassing volume (union) of all contoured volumes were considered as initial guesses. These preliminary tests showed that the resulting true volume is robust with respect to the initial guess and therefore the median value was used in the final version of the study. Furthermore, it should be mentioned that although only a single patient data was considered for each target case, the actual examples used in this study were chosen because they were representative for the types of targets used in radiosurgery. While more cases would add data to the study, the statistical power would not necessarily increase.

Since the STAPLE algorithm was introduced, it has been modified by the developers and other versions have been proposed (12-16). The Similarity and Truth Estimation for Propagated Segmentations (STEPS) algorithm (12) showed an improved performance in OAR segmentation (17) of head and neck cancer. An extended STAPLE algorithm was developed where the problem of quality and time for manual contouring was addressed (13). By the extension of the algorithm, partial contouring can be performed, either by experts contouring different structures or partial structures. Selective and iterative method for performance level estimation (SIMPLE) is another method that mainly differs from the STAPLE method by the feature that segmentations can be completely discarded in an iteration due to their bad performance, i.e. the algorithm can view a whole segmentation as an outlier (16), the authors therefore recommending this method to be applied to sub-volumes of the segmentations. Another method has been introduced that creates image segmentations from a single expert segmentation taking into account the image intensity and forcing certain areas to be included or excluded. These versions of the algorithm, however, appear to be similar with respect to determining the true volume if a number of contours delineated by qualified practitioners according to the local clinical practice is available, as in the case of this study, and therefore the initial STAPLE algorithm was considered to be an appropriate choice.

The robustness in target contouring could be ensured by applying a robust treatment planning optimization approach. This approach might overcome the need of considering several contours for the same target and determining the true volume. The robustness of treatment planning in general is usually ensured by accounting for errors (as, for example, the set-up errors in fractionated external beam radiotherapy), considering the organ motion and the potential anatomical changes as well as various uncertainties, for example, range uncertainties in proton therapy. Similarly, robust treatment planning optimization can be performed in terms of errors and uncertainties in the location and extent of the target volume. Furthermore, applying mathematical models for automatically producing plausible contours starting from a single delineation in a similar manner to the one proposed by Lê and colleagues (18) for rendering plausible image segmentation samples from a single expert segmentation could also be used for determining a probability distribution of the target boundaries to be further used in probabilistic planning optimisation. Thus, it is planned that the incorporation of the observed variability in the target contouring in radiosurgery, regarded as an uncertainty in the size and location of target, in a robust probabilistic treatment planning optimization approach, would be the natural continuation of this study.

Conclusion

If a high number of input contours is available, the STAPLE method provides a valuable tool in the estimation of the target true volume based on multiple contours as well as the sensitivity and specificity for each input contour relative to the true volume of that structure. The robustness of the STAPLE method for rendering the true target volume depends on the number of contours provided as input and their variability with respect to shape, size and position.

Acknowledgements

The cooperation of all participants involved in this study is gratefully acknowledged: Department of Stereotactic and Radiation Neurosurgery and Department of Medical Physics at Na Homolce Hospital in Prague, Czech Republic, Ian Paddick MSc and Tim Cox MD at Cromwell Gamma Knife Centre in London, England, Katherine Hunt BEng MSc MIPEM, Ruth Batty MD PhD FRCR, Dan Connolly MD PhD FRCR, John Yianni MD FRCS(SN) and Jeremy Rowe MD MA DM FRCS(SN) of the National Centre for Stereotactic Radiosurgery at Royal Hallamshire Hospital in Sheffield, England, Professor Lijun Ma and Professor Penny Sneed at the UCSF Gamma Knife Center in San Francisco, USA, Taylor McAdam Vell Neuroscience Institute at Washington Hospital in Fremont California, USA, Francisco Li MSc RSO at the Swedish Radiosurgery Center in Seattle, USA, Hyun-Tai Chung PhD at the Department of Neurosurgery Seoul National University College of Medicine in Seoul, Korea, Monique van Prooijen PhD at the Princess Margaret Cancer Centre in Toronto, Canada, Chryssa Paraskevopoulou MSc of Hygeia Hospital in Athens, Greece, Bente Sandvei Skeie MD PhD and Jan Heggdal MSc at the Department of Oncology and Medical Physics at Haukeland University Hospital in Bergen, Norway, David Schlesinger PhD, David Larson MD and Kevin Orcutt MD from the Department of Radiation Oncology at the University of Virginia in Charlottesville Virginia, USA. Technical assistance of Björn Somell at Elekta Instrument AB in Stockholm is also gratefully acknowledged.

Footnotes

  • Authors’ Contributions

    The study design was jointly proposed by Helena Sandström, Iuliana Toma-Dasu and Alexandru Dasu. Calculations were performed by Helena Sandström and Alexandru Dasu. Hidefumi Jokura provided patient data with input on study design. Caroline Chung gave input on study design and manuscript and Jonas Gårding provided consultance regarding the data analysis. The first draft of the manuscript was written by Helena Sandström and all Authors have reviewed the manuscript.

  • This article is freely accessible online.

  • Conflicts of Interest

    Jonas Gårding is employed by Elekta AB. The Authors have no other conflicts of interest to report.

  • Received October 19, 2020.
  • Revision received December 3, 2020.
  • Accepted December 4, 2020.

References

  1. ↵
    1. Sandström H,
    2. Nordström H,
    3. Johansson J,
    4. Kjäll P,
    5. Jokura H and
    6. Toma-Dasu I
    : Variability in target delineation for cavernous sinus meningioma and anaplastic astrocytoma in stereotactic radiosurgery with Leksell Gamma Knife Perfexion. Acta Neurochir (Wien) 156(12): 2303-2312, 2014. PMID: 25246145. DOI: 10.1007/s00701-014-2235-1
    OpenUrlCrossRefPubMed
    1. Sandström H,
    2. Chung C,
    3. Jokura H,
    4. Torrens M,
    5. Jaffray D and
    6. Toma-Dasu I
    : Assessment of organs-at-risk contouring practices in radiosurgery institutions around the world - The first initiative of the OAR Standardization Working Group. Radiother Oncol 121(2): 180-186, 2016. PMID: 27816408. DOI: 10.1016/j.radonc.2016.10.014
    OpenUrlCrossRefPubMed
  3. ↵
    1. Sandström H,
    2. Jokura H,
    3. Chung C and
    4. Toma-Dasu I
    : Multi-institutional study of the variability in target delineation for six targets commonly treated with radiosurgery. Acta Oncol 57(11): 1515-1520, 2018. PMID: 29786462. DOI: 10.1080/0284186X.2018.1473636
    OpenUrlCrossRefPubMed
  4. ↵
    1. Warfield SK,
    2. Zou KH and
    3. Wells W
    : Simultaneous truth and performance level estimation (STAPLE): an algorithm for the validation of image segmentation. IEEE Trans Med Imaging 23(7): 903-921, 2004. PMID: 15250643. DOI: 10.1109/TMI.2004.828354
    OpenUrlCrossRefPubMed
    1. Commowick O and
    2. Warfield SK
    : Estimation of inferential uncertainty in assessing expert segmentation performance from STAPLE. IEEE Trans Med Imaging 29(3): 771-780, 2010. PMID: 20199913. DOI: 10.1109/TMI.2009.2036011
    OpenUrlCrossRefPubMed
  6. ↵
    1. Hoang Duc AK,
    2. Eminowicz G,
    3. Mendes R,
    4. Wong S,
    5. McClelland J,
    6. Modat M,
    7. Cardoso MJ,
    8. Mendelson AF,
    9. Veiga C,
    10. Kadir T,
    11. D´Souza D and
    12. Ourselin S
    : Validation of clinical acceptability of an atlas-based segmentation algorithm for the delineation of organs at risk in head and neck cancer. Med Phys 42(9): 5027-5034, 2015. PMID: 26328953. DOI: 10.1118/1.4927567
    OpenUrlCrossRefPubMed
    1. Carles M,
    2. Fechter T,
    3. Nemer U,
    4. Nanko N,
    5. Mix M,
    6. Nestle U and
    7. Schaefer A
    : Feasibility of a semi-automated contrast-oriented algorithm for tumor segmentation in retrospectively gated PET images: phantom and clinical validation. Phys Med Biol 60(24): 9227-9251, 2015. PMID: 26576926. DOI: 10.1088/0031-9155/60/24/9227
    OpenUrlCrossRefPubMed
  8. ↵
    1. Eminowicz G and
    2. McCormack M
    : Variability of clinical target volume delineation for definitive radiotherapy in cervix cancer. Radiother Oncol 117(3): 542-547, 2015. PMID: 26475252. DOI: 10.1016/j.radonc.2015.10.007
    OpenUrlCrossRefPubMed
  9. ↵
    1. Cox BW,
    2. Spratt DE,
    3. Lovelock M,
    4. Bilsky MH,
    5. Lis E,
    6. Ryu S,
    7. Sheehan J,
    8. Gerszten PC,
    9. Chang E,
    10. Gibbs I,
    11. Soltys S,
    12. Sahgal A,
    13. Deasy J,
    14. Flickinger J,
    15. Quader M,
    16. Mindea S and
    17. Yamada Y
    : International spine radiosurgery consortium consensus guidelines for target volume definition in spinal stereotactic radiosurgery. Int J Radiat Oncol Biol Phys 83(5): e597-605, 2012. PMID: 22608954. DOI: 10.1016/j.ijrobp.2012.03.009
    OpenUrlCrossRefPubMed
  10. ↵
    1. Soliman H,
    2. Ruschin M,
    3. Angelov L,
    4. Brown PD,
    5. Chiang V,
    6. Kirkpatrick JP,
    7. Lo SS,
    8. Mahajan A,
    9. Oh K,
    10. Sheehan JP,
    11. Soltys SG and
    12. Sahgal A
    : Consensus contouring guidelines for postoperative completely resected cavity stereotactic radiosurgery for brain metastases. Int J Radiat Oncol Biol Phys 100(2): 436-442, 2018. PMID: 29157748. DOI: 10.1016/j.ijrobp.2017.09.047
    OpenUrlCrossRefPubMed
  11. ↵
    1. Segedin B and
    2. Petric P
    : Uncertainties in target volume delineation in radiotherapy – are they relevant and what can we do about them? Radiol Oncol 50(3): 254-262, 2016. PMID: 27679540. DOI: 10.1515/raon-2016-0023
    OpenUrlCrossRefPubMed
  12. ↵
    1. Jorge Cardoso M,
    2. Leung K,
    3. Modat M,
    4. Keihaninejad S,
    5. Cash D,
    6. Barnes J,
    7. Ourselin S and Alzheimer’s Disease Neuroimaging Initiative
    : STEPS: Similarity and truth estimation for propagated segmentations and its application to hippocampal segmentation and brain parcelation. Med Image Anal 17(6): 671-684, 2013. PMID: 23510558. DOI: 10.1016/j.media.2013.02.006
    OpenUrlCrossRefPubMed
  13. ↵
    1. Commonwick O and
    2. Warfield SK
    : Incorporating priors on expert performance parameters for segmentation validation and label fusion: a maximum a posteriori STAPLE. Med Image Comput Assist Interv 13(Pt 3): 25-32, 2010. PMID: 20879379. DOI: 10.1007/978-3-642-15711-0_4
    OpenUrlCrossRef
    1. Asman AJ and
    2. Landman BA
    : Robust statistical label fusion through consensus level, labeler accuracy, and truth estimation. IEEE Trans Med Imaging 30(10): 1779-1794, 2010. PMID: 21536519. DOI: 10.1109/TMI.2011.2147795
    OpenUrlCrossRef
    1. Commonwick O and
    2. Warfield SK
    : A continous STAPLE for scalar, vector and tensor images: An application to DTI analysis. IEEE Trans Med Imaging 28(6): 838-846, 2009. PMID: 19272988. DOI: 10.1109/TMI.2008.2010438
    OpenUrlCrossRefPubMed
  16. ↵
    1. Langerak TR,
    2. van der Heide UA,
    3. Kotte AN,
    4. Viergever MA,
    5. van Vulpen M and
    6. Pluim J
    : Label fusion in atlas-based segmentation using a selective and iterative method for performance level estimation (SIMPLE). IEEE Trans Med Imaging 29(12): 2000-2008, 2010. PMID: 20667809. DOI: 10.1109/TMI.2010.2057442
    OpenUrlCrossRefPubMed
  17. ↵
    1. Hoang duc AK,
    2. Eminowicz G,
    3. Mendes R,
    4. Wong S,
    5. McClelland J,
    6. Modat M,
    7. Cardoso MJ,
    8. Mendelson AF,
    9. Veiga C,
    10. Kadir T,
    11. D´Souza D and
    12. Ourselin S
    : Validation of clinical acceptability of an atlas-based segmentation algorithm for the delineation of organs at risk in head and neck cancer. Med Phys 42(9): 5027-5034, 2015. PMID: 26328953. DOI: 10.1118/1.4927567
    OpenUrlCrossRefPubMed
  18. ↵
    1. Lê M,
    2. Unkelbach J,
    3. Ayache N and
    4. Delingette H
    : Sampling image segmentations for uncertainty quantification. Med Image Anal 34: 42-51, 2016. PMID: 27198913. DOI: 10.1016/j.media.2016.04.005
    OpenUrlCrossRefPubMed
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools
Reprints and Permissions
Simultaneous Truth and Performance Level Estimation Method for Evaluation of Target Contouring in Radiosurgery
HELENA SANDSTRÖM, IULIANA TOMA-DASU, CAROLINE CHUNG, JONAS GÅRDING, HIDEFUMI JOKURA, ALEXANDRU DASU
Anticancer Research Jan 2021, 41 (1) 279-288; DOI: 10.21873/anticanres.14774
Twitter logo Facebook logo Mendeley logo

Jump to section

Keywords