Research ArticleExperimental Studies

Deep Learning-based Image Cytometry Using a Bit-pattern Kernel-filtering Algorithm to Avoid Multi-counted Cell Determination

TOMOKI ABE, KIMIHIRO YAMASHITA, TORU NAGASAKA, MITSUGU FUJITA, KYOUSUKE AGAWA, MASAYUKI ANDO, TOMOSUKE MUKOYAMA, KOTA YAMADA, SOUICHIRO MIYAKE, MASAFUMI SAITO, RYUICHIRO SAWADA, HIROSHI HASEGAWA, TAKERU MATSUDA, TAKASHI KATO, HITOSHI HARADA, NAOKI URAKAWA, HIRONOBU GOTO, SHINGO KANAJI, HIROAKI YANAGIMOTO, TARO OSHIKIRI, TETSUO AJIKI, TAKUMI FUKUMOTO and YOSHIHIRO KAKEJI
Anticancer Research August 2023, 43 (8) 3755-3761; DOI: https://doi.org/10.21873/anticanres.16560

Abstract

Background/Aim: In pathology, the digitization of tissue slide images and the development of image analysis by deep learning have dramatically increased the amount of information obtainable from tissue slides. This advancement is anticipated to not only aid in pathological diagnosis, but also to enhance patient management. Deep learning-based image cytometry (DL-IC) is a technique that plays a pivotal role in this process, enabling cell identification and counting with precision. Accurate cell determination is essential when using this technique. Herein, we aimed to evaluate the performance of our DL-IC in cell identification. Materials and Methods: Cu-Cyto, a DL-IC with a bit-pattern kernel-filtering algorithm designed to help avoid multi-counted cell determination, was developed and evaluated for performance using tumor tissue slide images with immunohistochemical staining (IHC). Results: The performances of three versions of Cu-Cyto were evaluated according to their learning stages. In the early stage of learning, the F1 score for immunostained CD8+ T cells (0.343) was higher than the scores for non-immunostained cells [adenocarcinoma cells (0.040) and lymphocytes (0.002)]. As training and validation progressed, the F1 scores for all cells improved. In the latest stage of learning, the F1 scores for adenocarcinoma cells, lymphocytes, and CD8+ T cells were 0.589, 0.889, and 0.911, respectively. Conclusion: Cu-Cyto demonstrated good performance in cell determination. IHC can boost learning efficiencies in the early stages of learning. Its performance is expected to improve even further with continuous learning, and the DL-IC can contribute to the implementation of precision oncology.

Key Words:

In recent years, the accuracy of image recognition using deep learning has dramatically improved and is expected to find applications in the medical field. The digitization of whole-slide images (WSIs) of tissues and the emergence of new high-intensity, high-dimensional, and high-resolution technologies, has enabled the acquisition and analysis of large volumes of biological information obtained from tissue slides (1). For example, deep learning methods using hematoxylin and eosin (H&E)-stained slides to analyze tumor tissues are being used by pathologists and have attracted a great deal of attention, not only because they allow for automation and standardization of pathological features (2), but also because they may be helpful for clinical patient management (3).

Deep learning-based image cytometry (DL-IC) is a technological platform that automatically determines the type and quantity of cells by analyzing microscopic images using deep learning methods to recognize each cell morphologically and capture labeled molecules localized both inside and outside of it (4, 5). Approaches to the automated analysis of microscopic data originated in the 1950s, and since then, various methods have been developed and studied for detecting specific cellular structures, such as nuclei, and extracting phenotypic characteristics (6). As time has progressed, the accuracy of automated cell counting has become a crucial factor in assessing the clinical value of these methods. One method for identifying individual cells is to divide tissue slide images into grid patterns and perform cell determination for each partial image (7). When using this method, it is necessary to avoid over-counting due to multi-counted cell determination, wherein one cell is counted multiple times, and under-counting, wherein overlapping cells are not separated.

We developed an algorithm to overcome the over-counting problem and successfully constructed a DL-IC with this algorithm. In this study, we evaluated its performance using immuno-stained images.

Materials and Methods

Study population. This study included 68 patients who underwent radical surgery for primary colorectal cancer at the Kobe University Hospital (Kobe, Japan) between January 2005 and December 2016. Only patients who met the following three conditions were included in the analysis: 1) cases pathologically confirmed as adenocarcinoma; 2) those with clinical T3-4 or clinical N (+), H0P0M0; and 3) cases with or without preoperative treatment, for whom tissue collection was available. We determined tumor-node-metastasis (TNM) classifications according to Union for International Cancer Control (UICC) 7th edition (8).

Whole-slide images and patch extraction. The procedure used in this study has been described previously (9). Briefly, H&E and immunohistochemical stainings (IHC) were performed on 4-μm slices. The slides were de-paraffinized with xylene and rehydrated using graded dilutions of ethanol. Antigen retrieval was performed by heating the samples in an oven. The slides were stained with H&E and immune-stained with an Anti-CD8 mouse monoclonal antibody (Dako, C8/144B). The slides were then digitized as WSIs using a Hamamatsu NDP slide scanner, Hamamatsu Nanozoomer 2.0 HT (Hamamatsu Photonics K.K. Hamamatsu, Japan) and viewed using the NDP viewer platform (Hamamatsu Photonics K.K.). A typical slide image is shown in Figure 1.

Figure 1.
Figure 1.

Tissue slide images with immunohistochemical staining. A tissue slide image of rectal cancer was stained with H&E and immune-stained with an anti-CD8 antibody. Black arrows indicate cancer cells, and white arrows indicate lymphocytes. Immunostained red cells were CD8+ T cells.

Bit-pattern kernel-filtering algorithm. WSIs were scanned at 20× magnification (443 nm/pixel) and sequentially cropped into 18.3×18.3 μm grid-like partial images. As shown in Figure 2A, these images were then analyzed using a DL-IC composed of convolutional neural networks or residual networks (ResNet) (10). Due to the grid size being smaller than the nuclei of most cell types, instances of under-counting cells were infrequent. For example, as shown in Figure 2A, it was possible to identify four biological cells within a single square of the grid (enclosed by the white dotted line). The cell determination grid points were converted to ON (denoted as 1) or OFF (denoted as 0) pixels (Figure 2B). Adjacent ON pixels in the vertical, horizontal, and diagonal directions were found to be responsible for multi-counting of cells. An algorithm for combining these adjacent ON pixels into a single pixel was, therefore, developed as follows: a scan with bit-pattern kernel-filtering was performed on the input matrix, and if there was a matched pattern, the pixel at the center of the site was recorded as an ON pixel (Figure 3A). Different types of bit-pattern kernels were identified based on the number of ON pixels in a 3×3-pixel grid, some of which are shown in Figure 3B. Each step consisted of combining several kernels. The adjacent ON pixels were removed by processing the images sequentially in several steps. All 3×3-pixels were processed by the bit-pattern kernel-filtering algorithm until all 3×3 pixels had been reduced to Kernel 1. The process of removing adjacent ON pixels using this bit-pattern kernel-filtering algorithm is illustrated in Figure 3C. The details of this algorithm have been patented (PCT/JP2017/024994). An example of the removal of adjacent ON pixels and the avoidance of multi-counted cell determination is shown in Figure 3D and E.

Figure 2.
Figure 2.

Cell determination using a deep learning-based model and tissue slide images. (A) A tissue slide image was divided into a grid, and cell determination was performed by the deep learning-based model. As the cells were enclosed by squares bound by the white dotted lines, it was possible to make four cell determinations within a single square. (B) For cell determination, tissue slide images were converted to a matrix image of ON or OFF pixels. The cell determination grid point was converted to ON (denoted as 1) or OFF pixels (denoted as 0).

Figure 3.
Figure 3.
Figure 3.

Avoiding multi-counted cell determination using a bit-pattern kernel-filtering algorithm. (A) An example of removing adjacent ON pixels ON pixels horizontally, vertically, and diagonally adjacent to the central ON pixel are removed, resulting in a single central pixel. (B) Types of bit-pattern kernels for removing adjacent ON pixels. One type of 3×3-pixels with one ON pixel (Kernel 1), 8 types of Kernel 2, 20 types of Kernel 3, 32 types of Kernel 4 (figure not shown), and 38 types of Kernel 5 (figure not shown) were set. (C) Each filtering step consisting of a combination of several Kernels. By processing images one after another in several steps, adjacent ON pixels are integrated. All 3×3-pixels are processed by the bit-pattern kernel-filtering algorithm until all 3×3 pixels become Kernel 1. (D) Output matrix after passing through each step. At each step, adjacent ON pixels are removed. All 3×3-pixels were reduced to Kernel 1, with no more adjacent ON pixels in the final output. (E) An example of avoiding multi-counted cell determination in a tissue slide image. Pins represent cell determination grid points. Before applying the algorithm, multi-pins (white arrow) are placed on one cell (enclosed by the white dotted lines). After applying the algorithm, multi-counted cell determinations are avoided.

Deep learning-based image cytometry applied in training and validation sets. A DL-IC, Cu-Cyto, was developed by combining a deep learning-based method with our bit-pattern kernel-filtering algorithm (https://www.ai-and-eye.co.jp/). Cu-Cyto can detect nearly 20 different cell types (e.g., adenocarcinoma cells, lymphocytes, normal glandular epithelium cells, stromal cells, and macrophages).

Cell determination was performed using each version of Cu-Cyto, according to the three learning stages. To train and validate the DL-IC, seven well-trained investigators, including one pathologist, annotated the cells. A total of 25,771 samples of adenocarcinoma cells, 33,240 samples of lymphocytes, 8,227 samples of CD8+ T cells, and 65,448 samples of other cells (such as stromal cells, macrophages, and others) were collected. Of these, 512 (29) samples of adenocarcinoma cells, lymphocytes, and CD8+ T cells, and 5,466 samples of other cells were used for the training and validation sets of Cu-Cyto version (v)1. For Cu-Cyto v2, 2,048 (211) samples of adenocarcinoma cells, lymphocytes, and CD8+ T cells, and 15,957 samples of other cells were used for the training and validation sets. For Cu-Cyto v3, 8,192 (213) samples of adenocarcinoma cells, lymphocytes, and CD8+ T cells and 37,530 samples of other cells were used for the training and validation sets. This study aimed to evaluate the performance of Cu-Cyto for the determination of adenocarcinoma cells, lymphocytes, and CD8+ T cells.

Performance evaluation of Cu-cyto. As a test set, we evaluated the performances of three versions of the Cu-Cyto (v1, v2, and v3). Thirty slides containing 893 adenocarcinoma cells, 2,515 lymphocytes, 689 CD8+ T cells, and 2,449 cells of other types were analyzed using Cu-Cyto.

To assess the performance of Cu-Cyto, we compared its results with those obtained by an experienced pathologist. Performance metrics were calculated using the numbers of true positives (TPs), true negatives (TNs), false negatives (FNs), and false positives (FPs). The formulas were as follows: Accuracy=(TP+TN)/(TP+TN+FN+FP); Sensitivity=TP/(TP+FN); Specificity=TN/(TN+FP); Positive predictive value (PPV)=TP/(TP+FP); Negative predictive value (NPV)=TN/(TN+FN); and F1 score=2×Sensitivity ×PPV/(Sensitivity+PPV).

Results

Performance evaluation of Cu-cyto. Figure 4 shows the progression of the F1 score, a commonly used metric for evaluating cell determination performance. In Cu-Cyto v1, the F1 score for immunostained CD8+ T cells surpassed that of non-immunostained cells (adenocarcinoma cells and lymphocytes). With ongoing training and validation, the F1 scores improved for all cell types. In Cu-Cyto v3, the F1 scores for CD8+ T cells and lymphocytes were comparable, reaching approximately 0.9. The latest results are summarized in Table I. Both adenocarcinoma cells and CD8+ T cells showed comparable levels of performance improvement between v1 and v3. However, lymphocytes showed a significant improvement only between v2 and v3.

Figure 4.
Figure 4.

Performance evaluation of Cu-Cyto. The performance of Cu-Cyto was evaluated using F1 scores. In Cu-Cyto version (v)1, the F1 scores for adenocarcinoma cells, lymphocytes, and CD8+ T cells were 0.040, 0.002, and 0.343, respectively. In Cu-Cyto v2, the F1 scores for adenocarcinoma cells, lymphocytes, and CD8+ T cells were 0.300, 0.047, and 0.634, respectively. In Cu-Cyto v3, the F1 scores for adenocarcinoma cells, lymphocytes, and CD8+ T cells were 0.589, 0.889, and 0.911, respectively. F1 score=2×Sensitivity×PPV/ (Sensitivity+PPV). PPV, Positive predictive value.

View this table:
Table I.

Performance of Cu-Cyto for cell determination.

Discussion

In this study, we constructed Cu-Cyto, a DL-IC consisting of a deep learning-based method and an algorithm to avoid multi-counted cell determination. In the performance analysis of Cu-Cyto, its cell determination ability was found to be good.

We found that the cell determination performance of Cu-Cyto was higher for immunostained cells than for non-immunostained cells in the early stages of the training set. Our results suggest that IHC can boost the learning efficiency of deep learning-based models, especially in the early stages of learning. Learning efficiency varied according to cell type, with lymphocytes showing the most rapid performance improvement. Combining multiplex staining is predicted to improve the accuracy of the analysis (11), but the high cost of this method is an issue. The DL-IC with appropriate IHC is superior for analyzing large numbers of tissue slides with reduced labor and cost. This method is expected to be useful for extracting information on specific cells from extensive clinical trial data and for pre-analytical screening that uses many parameters.

Tumor tissues in the human body form complex ecosystems known as tumor immune microenvironments (TIMEs), which are composed of not only tumor cells, but also immune cells, stromal cells, and various other cell types, as well as surrounding extracellular matrix proteins and soluble components (12). TIME has attracted much attention for its role in tumor biology and its clinical relevance to patient outcomes and therapeutic efficacy in precision oncology (13). To describe these complex TIME characteristics, technologies have been developed to analyze and capture information on not only the density and phenotype of cells, but also the spatial structures of TIMEs, including the distance between various cells and unique structures, using tumor tissue slides (14). Cu-Cyto is one of these technologies, for which there are high hopes in the future. Data quality control is essential to ensure that TIME-related biometric information is clinically applicable. The accuracy of the input data is a cornerstone of these technologies. Thus, the developed filtering algorithm is becoming increasingly crucial for data management.

There are several limitations to this study. First, it showed the performance of cell determination for 3 cell types of, but the accuracy is insufficient. Second, although the current model can determine more than 20 cell types, it achieves high accuracy only for certain cell types, thus we present only a subset of the data. However, learning for this model is ongoing, and further performance improvements are expected. We predict that Cu-Cyto’s F1 scores for the determination of each cell type will exceed 0.9 after 30,000 (215) training instances for each cell types, and the determinable cell number will increase with further learning. Third, to demonstrate the superiority of a DL-IC with the bit-pattern kernel-filtering algorithm, it is necessary to compare it with a DL-IC without this algorithm, or with the bounding box method (15), which is another standard method for object recognition. We are going to conduct comparisons in the future.

In this study, we showed that the Cu-Cyto, with our newly-developed bit-pattern kernel-filtering method, has good cell determination ability. By applying multiple IHC to the current DL-IC, we will develop a model with higher discrimination accuracy. For further detailed analysis, we will develop a DL-IC that can obtain spatial information that requires higher dimensional discrimination, such as the distances between various cells that make up the TIMEs and structures with unique patterns. Finally, we will perform a detailed analysis of large samples, such as in large-scale clinical trials.

Acknowledgements

This research was supported by JSPS KAKENHI Grant Number JP 20H03752 (Y.K.), 22K08892 (T.O.), 21K08778 (T.M.) and 22K15606 (R.S.)

Footnotes

  • Authors’ Contributions

    TA, KY, TN, and MF conceived the study. KA, MA, TM, KY, SM, MA, RS, HH, TM, TK, HH, NU, HG, SK, HK, TO, and TA accumulated data. TF and YK supervised the conduct of this study. All Authors reviewed the manuscript draft and approved the final version for submission.

  • Conflicts of Interest

    The Authors declare that they have no conflicts of interest.

  • Received May 13, 2023.
  • Revision received June 16, 2023.
  • Accepted June 19, 2023.

References

  1. ↵
    1. Fu Y,
    2. Jung A,
    3. Torne R,
    4. Gonzalez S,
    5. Vöhringer H,
    6. Shmatko A,
    7. Yates L,
    8. Jimenez-Linan M,
    9. Moore L,
    10. Gerstung M
    : Pan-cancer computational histopathology reveals mutations, tumor composition and prognosis. Nat Cancer 1(8): 800-810, 2020. DOI: 10.1038/s43018-020-0085-8
    OpenUrlCrossRefPubMed
  2. ↵
    1. Kanavati F,
    2. Tsuneki M
    : Breast invasive ductal carcinoma classification on whole slide images with weakly-supervised and transfer learning. Cancers 13(21): 5368, 2021. DOI: 10.3390/cancers13215368
    OpenUrlCrossRefPubMed
  3. ↵
    1. Saltz J,
    2. Gupta R,
    3. Hou L,
    4. Kurc T,
    5. Singh P,
    6. Nguyen V,
    7. Samaras D,
    8. Shroyer KR,
    9. Zhao T,
    10. Batiste R,
    11. Van Arnam J, Cancer Genome Atlas Research Network,
    12. Shmulevich I,
    13. Rao A,
    14. Lazar A,
    15. Sharma A,
    16. Thorsson V
    : Spatial organization and molecular correlation of tumor-infiltrating lymphocytes using deep learning on pathology images. Cell Rep 23: 181-193.e7, 2018. DOI: 10.1016/j.celrep.2018.03.086
    OpenUrlCrossRefPubMed
  4. ↵
    1. Gupta A,
    2. Harrison P,
    3. Wieslander H,
    4. Pielawski N,
    5. Kartasalo K,
    6. Partel G,
    7. Solorzano L,
    8. Suveer A,
    9. Klemm A,
    10. Spjuth O,
    11. Sintorn I,
    12. Wählby C
    : Deep learning in image cytometry: a review. Cytometry A 95(4): 366-380, 2019. DOI: 10.1002/cyto.a.23701
    OpenUrlCrossRef
  5. ↵
    1. Moen E,
    2. Bannon D,
    3. Kudo T,
    4. Graf W,
    5. Covert M,
    6. Van Valen D
    : Deep learning for cellular image analysis. Nat Methods 16(12): 1233-1246, 2019. DOI: 10.1038/s41592-019-0403-1
    OpenUrlCrossRefPubMed
  6. ↵
    1. Kraus O,
    2. Grys B,
    3. Ba J,
    4. Chong Y,
    5. Frey B,
    6. Boone C,
    7. Andrews B
    : Automated analysis of high-content microscopy data with deep learning. Mol Syst Biol 13(4): 924, 2017. DOI: 10.15252/msb.20177551
    OpenUrlAbstract/FREE Full Text
  7. ↵
    1. Bao G,
    2. Wang X,
    3. Xu R,
    4. Loh C,
    5. Adeyinka O,
    6. Pieris D,
    7. Cherepanoff S,
    8. Gracie G,
    9. Lee M,
    10. McDonald K,
    11. Nowak A,
    12. Banati R,
    13. Buckland M,
    14. Graeber M
    : PathoFusion: an open-source AI framework for recognition of pathomorphological features and mapping of immunohistochemical data. Cancers 13(4): 617, 2021. DOI: 10.3390/cancers13040617
    OpenUrlCrossRefPubMed
  8. ↵
    1. Sobin LH,
    2. Gospodarowicz MK,
    3. Wittekind C
    : TNM Classification of Malignant Tumours, 7th Edition. Wiley-Blackwell, 2009.
  9. ↵
    1. Agawa K,
    2. Yamashita K,
    3. Nakagawa A,
    4. Yamada K,
    5. Watanabe A,
    6. Mukohyama J,
    7. Saito M,
    8. Fujita M,
    9. Takiguchi G,
    10. Urakawa N,
    11. Hasegawa H,
    12. Kanaji S,
    13. Matsuda T,
    14. Oshikiri T,
    15. Nakamura T,
    16. Suzuki S,
    17. Kakeji Y
    : Simple cancer stem cell markers predict neoadjuvant chemotherapy resistance of esophageal squamous cell carcinoma. Anticancer Res 41(8): 4117-4126, 2021. DOI: 10.21873/anticanres.15214
    OpenUrlAbstract/FREE Full Text
  10. ↵
    1. He K,
    2. Zhang X,
    3. Ren S,
    4. Sun J
    : Deep residual learning for image recognition. arXiv: 770-778, 2016.
  11. ↵
    1. Angelo M,
    2. Bendall S,
    3. Finck R,
    4. Hale M,
    5. Hitzman C,
    6. Borowsky A,
    7. Levenson R,
    8. Lowe J,
    9. Liu S,
    10. Zhao S,
    11. Natkunam Y,
    12. Nolan G
    : Multiplexed ion beam imaging of human breast tumors. Nat Med 20(4): 436-442, 2014. DOI: 10.1038/nm.3488
    OpenUrlCrossRefPubMed
  12. ↵
    1. Pitt J,
    2. Marabelle A,
    3. Eggermont A,
    4. Soria J,
    5. Kroemer G,
    6. Zitvogel L
    : Targeting the tumor microenvironment: removing obstruction to anticancer immune responses and immunotherapy. Ann Oncol 27(8): 1482-1492, 2016. DOI: 10.1093/annonc/mdw168
    OpenUrlCrossRefPubMed
  13. ↵
    1. Fukuoka E,
    2. Yamashita K,
    3. Tanaka T,
    4. Sawada R,
    5. Sugita Y,
    6. Arimoto A,
    7. Fujita M,
    8. Takiguchi G,
    9. Matsuda T,
    10. Oshikiri T,
    11. Nakamura T,
    12. Suzuki S,
    13. Kakeji Y
    : Neoadjuvant chemotherapy increases PD-L1 expression and CD8+ tumor-infiltrating lymphocytes in esophageal squamous cell carcinoma. Anticancer Res 39(8): 4539-4548, 2019. DOI: 10.21873/anticanres.13631
    OpenUrlAbstract/FREE Full Text
  14. ↵
    1. Fu T,
    2. Dai L,
    3. Wu S,
    4. Xiao Y,
    5. Ma D,
    6. Jiang Y,
    7. Shao Z
    : Spatial architecture of the immune microenvironment orchestrates tumor immunity and therapeutic response. J Hematol Oncol 14(1): 98, 2021. DOI: 10.1186/s13045-021-01103-4
    OpenUrlCrossRef
  15. ↵
    1. Redmon J,
    2. Divvala S,
    3. Girshick R,
    4. Farhadi A
    : You only look once: Unified, real-time object detection. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. DOI: 10.1109/CVPR.2016.91
    OpenUrlCrossRef
Back to top

In this issue

Print
Download PDF
Article Alerts
Email Article
Citation Tools
Reprints and Permissions
Deep Learning-based Image Cytometry Using a Bit-pattern Kernel-filtering Algorithm to Avoid Multi-counted Cell Determination
TOMOKI ABE, KIMIHIRO YAMASHITA, TORU NAGASAKA, MITSUGU FUJITA, KYOUSUKE AGAWA, MASAYUKI ANDO, TOMOSUKE MUKOYAMA, KOTA YAMADA, SOUICHIRO MIYAKE, MASAFUMI SAITO, RYUICHIRO SAWADA, HIROSHI HASEGAWA, TAKERU MATSUDA, TAKASHI KATO, HITOSHI HARADA, NAOKI URAKAWA, HIRONOBU GOTO, SHINGO KANAJI, HIROAKI YANAGIMOTO, TARO OSHIKIRI, TETSUO AJIKI, TAKUMI FUKUMOTO, YOSHIHIRO KAKEJI
Anticancer Research Aug 2023, 43 (8) 3755-3761; DOI: 10.21873/anticanres.16560
Twitter logo Facebook logo Mendeley logo

Jump to section

Keywords