Abstract
Background/Aim: Laparoscopic hepatectomy (LH) requires accurate visualization and appropriate handling of hepatic veins and the Glissonean pedicle that suddenly appear during liver dissection. Failure to recognize these structures can cause injury, resulting in severe bleeding and bile leakage. This study aimed to develop a novel artificial intelligence (AI) system that assists in the visual recognition and color presentation of tubular structures to correct the recognition gap among surgeons. Patients and Methods: Annotations were performed on over 350 video frames capturing LH, after which a deep learning model was developed. The performance of the AI was evaluated quantitatively using intersection over union (IoU) and Dice coefficients, as well as qualitatively using a two-item questionnaire on sensitivity and misrecognition completed by 10 hepatobiliary surgeons. The usefulness of AI in medical education was qualitatively evaluated by 10 medical students and residents. Results: The AI model was able to individually recognize and colorize hepatic veins and the Glissonean pedicle in real time. The IoU and Dice coefficients were 0.42 and 0.53, respectively. Surgeons provided a mean sensitivity score of 4.24±0.89 (from 1 to 5; Excellent) and a mean misrecognition score of 0.12±0.33 (from 0 to 4; Fail). Medical students and residents assessed the AI to be very useful (mean usefulness score, 1.86±0.35; from 0 to 2; Excellent). Conclusion: The novel AI presented was able to assist surgeons in the intraoperative recognition of microstructures and address the recognition gap among surgeons to ensure a safer and more accurate LH.
The liver is a complex organ containing various vessels, bile ducts, and portal veins, which vary among each individual. Therefore, liver resection requires accurate preoperative comprehension of the anatomy and correct intraoperative structural recognition. One study reported on the use of intraoperative ultrasound for liver resection (1), whereas another study proposed injection of a dye, such as indigo carmine, into the portal vein branch, which is the boundary of the liver segment, to describe the demarcation line during anatomical hepatectomy (2). Since the publication of our indocyanine green (ICG) fluorescence method for liver resection in 2008 (3), it has been widely used for real-time navigation, especially for liver segment staining and tumor identification. However, no changes in complication rates for liver surgery have been noted. This is because surgical progress and decision-making depend on the surgeon’s ability to recognize anatomical structures (4, 5). In particular, hepatic veins and the Glissonean pedicle that suddenly appear in the anatomical transection plane of the liver are easily misidentified given their similar color to surrounding tissues and are often damaged and improperly treated, resulting in bleeding and bile leakage. To address these concerns, a new artificial intelligence (AI) model was developed to provide visual assistance to surgeons by automatically recognizing and coloring tubular structures that appear during laparoscopic hepatectomy (LH). AI-based assistance in anatomical structure recognition is expected to support the surgeons’ skills (6) and be used as a decision-making tool for surgical progress.
Patients and Methods
No AI tools were used for the writing of the manuscript, production of images or graphical elements of the paper, or in the collection and analysis of data.
Study design and datasets. This single-institution study collected 10 videos on LH at our Institution from 2020 to 2022, which were used to develop and evaluate an AI algorithm that aims to recognize the hepatic vein and Glissonean pedicle and display them in color during liver dissection. Each surgical video was produced at 30 frames per second (fps) and a display resolution of 1,980×1,280 pixels as preprocessing. The videos were selected from a variety of procedures, including partial resection, cone unit resection, anatomical subsegmental resection, and left lateral resection. To facilitate annotation of tubular structures, such as the hepatic vein and Glissonean pedicle, videos in which the tubular structures were depicted in the liver dissection plane were selected. Ultimately, 10 eligible surgical videos were clipped and downloaded to a hard drive. Given that the videos were anonymous, no demographic or personal identifiers were attached to this dataset.
Annotations. A total of 350 static images were extracted from each video with clearly depicted tubular structures in the liver dissection plane, after which they were framed and saved in PNG format at a resolution of 1,920×1,080 pixels (aspect ratio 16:9). The boundaries and surface of the hepatic vein and Glissonean pedicle, whatever the size, were precisely annotated on each frame by four surgeons specializing in gastro-enterology and hepatobiliary pancreatic surgery and have adequate experience in both laparoscopic and laparotomy hepatectomy to create the training set. The neural network model was based on the DeepLab v3 architecture, which has previously shown promising results, especially in medical image segmentation tasks. Deep learning algorithms enable a more accurate segmentation map output by extracting object features in the convolution layer while recovering location information in the deconvolution layer. Model training and inference were performed on a workstation with a NVIDIA A40 GDDR6 (NVIDIA Corp., Santa Clara, CA, USA) with 48 GB of memory. The static image was enlarged on the screen, and the surface features of the hepatic vein and Glissonean pedicle were carefully and delicately colored and annotated in detail. The hepatic vein and Glissonean pedicle region were highlighted in blue and green, respectively, to produce automatic segmentation results at more than 30 fps.
Development of the AI model. The prototype AI model in the present study was produced in 2022 using 200 static images taken from 5 training videos using the DeepLab v3 architecture developed by augmenting the training data with surgeons’ annotations. The process of updating the prototype AI model was conducted by through more advanced annotation and data augmentation while keeping the DeepLab v3 architecture intact. A total of 350 images extracted from a total of 10 training videos and annotations enforced were used to train the latest AI model (Figure 1).
Development of an artificial intelligence system and evaluation of its performance. AI: Artificial intelligence, LH: Laparoscopic hepatectomy, HBP: Hepato–biliary–pancreatic.
Preoperative resection planning under three-dimensional simulation. Laparoscopic liver resection requires careful identification of the hepatic vein, Glissonean pedicle, and tumor location and their relationship with each other before surgery. Preoperative computed tomography (CT) using a 64-row multidetector CT scanner was performed for all patients who underwent LH. Preoperative surgical planning was performed using Synapse Vincent software (Fujifilm Medical, Tokyo, Japan) and Revoras software (Ziosoft, Tokyo, Japan) by hepatopancreatobiliary (HBP) surgeons to obtain information on the anatomical structures. We routinely construct three-dimensional simulations and rehearse surgical strategies in advance using this software. These preoperative simulations enable surgeons to review the reconstructed liver structures (i.e., liver parenchyma, portal veins, hepatic veins, and tumors), establish the liver segmentation, and determine the correct transection plane for the surgery. Thereafter, the surgeons ascertain the patient-specific liver anatomy and relationship between vessels and tumors to ensure an accurate and safe liver resection. Furthermore, surgeons can determine the vascular structures that will appear on the expected liver transection plane, as well as the main and important vessels or Glissonean pedicle that should be treated during the surgery. These meticulous and detailed surgical planning ensures accurate and safe surgeries.
Evaluation of AI model performance by computation. The surgeons randomly selected frames from the videos that underwent tubular structure prediction by the latest AI. The annotators manually segmented the corresponding frames from the original videos to create the ground truth. To evaluate the accuracy at which the AI recognized the hepatic vein and Glissonean pedicle, the Dice coefficient (DC) was determined for each image and was used as the evaluation metric in the present study. This coefficient indicates the degree of match between the segmentation area of the deep learning model and manual annotation and can range from zero to one, with one indicating perfect AI prediction and zero indicating no prediction pixel overlap with the ground truth area.
The intersection over unit (IoU), which is essentially a method of quantifying the overlap between the AI prediction area and ground truth area, was also calculated to evaluate the AI prediction accuracy. The IoU is calculated using the area of overlap and union (i.e., the sum of AI prediction and ground truth). DC and IoU have been widely used performance metrics for assessing sensitivity in machine learning. IoU and DC were defined as follows:
IoU=Area of overlap/Area of Sum set of AI and Ground truth
TP: True positive; FP: false positive; FN: false negative.
Evaluation of AI model performance by trained surgeons. During quantitative assessments, clinicians can find it difficult to interpret the values and determine their validity for clinical application, especially in the case of visual and cognitive assessments. Accordingly, a questionnaire was developed with reference to previous studies for the purpose of complementing the quantitative evaluation. A total of 60 static images with clearly depicted vascular structures were extracted from the LH videos analyzed by the AI. Thereafter, 10 HBP surgeons who routinely perform surgeries completed a questionnaire to qualitatively evaluate the performance of the AI model (Figure 2). The images used for the questionnaire were displayed together with the original unanalyzed static images, and the evaluators answered the questionnaire intuitively. Scoring was left to the discretion of each surgeon (Figure 1).
Qualitative evaluation of AI model performance. Q: Question.
The questions were as follows:
Q1. How sensitive was the AI in recognizing tubular structures on the liver transection plane?
The average value of each frame was used as the sensitivity score, which was evaluated in five steps of 20% increments (1: 0%-19% recognition rate, 5: 80%-100% recognition rate).
Q2. How many tubular structures did the AI unrecognized or misrecognized?
The average value of each frame was used as the misrecognition (MR) score, which was evaluated in five steps (0: No misrecognized tubular structures, 4: 4 or more misrecognized structures)
Evaluation of AI model performance by medical students and residents. Medical students and residents have difficulty recognizing anatomical structures given their lack of education on the microanatomy of the liver.
AI has been used during LH surgery for the purpose of medical education, particularly with helping medical students and residents efficiently understand the surgical procedure and familiarize with what is displayed on the monitor during their surgical observation. Here, medical students and residents were asked whether the AI system was useful for their anatomical understanding (Figure 2). As such, 10 static images were extracted from the LH video, showing the original frames followed by AI-enhanced frames. They were then asked to evaluate whether the tubular structures on the liver transection plane highlighted by the AI would assist them in their recognition and learning (Figure 1).
Q3. Was the AI useful with respect to intraoperative structure recognition?
Medical students and residents rated usefulness based on a 3-point scale (2: Useful for recognition assistance, 1: Interim valuation, 0: Not useful).
Furthermore, for the purpose of evaluating the speed and accuracy of AI analysis, an experimental study was conducted to analyze surgical videos in real time via a connection to a surgical monitor that is actually being used in the operating room. The monitor displaying the AI analysis images was installed outside the operating room, setting up an environment that would not affect the surgeons performing the surgery. We also assessed the possibility of using surgical endoscopic systems from different companies, namely 1588 AIM (Stryker, Kalamazoo, MI, USA), VISERA ELITE II (Olympus, Bartlett, TN, USA), VISERA ELITE III (Olympus), for improving recognition.
Statistical analysis. Data are presented as mean±standard deviation, unless otherwise specified. All analyses were performed using JMP Pro 15 (SAS Institute Inc., Cary, NC, USA).
Ethics approval. The study protocol was approved by ethics committee of our institution (approval number: 21-103-B).
Results
The AI system constructed herein provided a color-coded display of the tubular structures appearing on the liver transection plane during LH, with the hepatic veins and Glissonean pedicle displayed in blue and green, respectively. The original and AI-enhanced images presented in Figure 3. It is clear that the coloring is displayed very delicately, softly, and without loss of three-dimensionality (Figure 3a and b). Even in the presence of ICG fluorescence, the structures highlighted by the AI were sharp and clear (Figure 3c and d). Real-time analysis of the surgical video was possible, and the enhancement of the tubular structures was exceedingly clear. The mean time gap between the surgical video and the analysis display was 0.118±0.018 s. The veins and Glissonean pedicle were clearly color-coded and displayed in blue and green, respectively, using the AI.
Artificial intelligence system prediction results with high sensitivity. (a, c) An original frame of cases 1 and 2. (b, d) Prediction by the AI model in cases 1 and 2 (White arrow: tubular structure). IoU: Intersection over union; DC: dice coefficients.
With regard to the quantitative evaluation of the AI, the average DC and IoU were 0.53 and 0.42, respectively, demonstrating acceptable sensitivity and similarity between automatic and manual segmentation.
AI performance metrics and qualitative scores measured by HBP surgeons in Q1 and Q2 are described in Table I. Notably, the mean sensitivity score was high (4.24; range=2.80-5.00), whereas the mean MR score was quite low (0.12; range=0-0.60), suggesting that the accuracy at which the AI recognized the tubular structures was high, with little under- or misrecognition.
Qualitative scores (mean, SD) in 60 randomly sampled video frames.
The usefulness of AI as a medical education tool evaluated by medical students and residents in Q3 is described in Table II. Notably, medical students and residents provided the AI with a high usefulness score (1.80). Moreover, 86% of all respondents answered Q3 as useful (score of 2), whereas none of the respondents answered that it had no impact on recognition or that it was not useful (score of 0).
Usefulness scores (mean, SD) in 10 AI analyzed still frames.
Figure 4 demonstrates the relationships between the sensitivity score and MR scores for each frame with Q1 and Q2 for further analysis. Figure 4a describes a mosaic diagram created based on Q1 and Q2, including all answers for 60 frames by 10 surgeons. The most common combination of two answers for Q1 and Q2 was score 5 for Q1 and score 0 for Q2 (47.7% of all answers), followed by score 4 and score 0 (21.8%), score 3 and score 0 (15.7%). Figure 4b presents a scatter plot that demonstrates the relationship between the sensitivity score and MR scores for each frame. The 95% probability, represented by a blue ellipse, was concentrated in the upper left portion of the coordinates due to the high sensitivity score and the relatively low MR scores for each frame.
Relationship among the qualitative scores. (a) A mosaic diagram showing the distribution of all scores assigned by hepatopancreatobiliary surgeons. Vertical and horizontal axes represent the proportion of scores assigned to Q1 and Q2, respectively. (b) Scatter plot showing the relation between sensitivity score and misrecognition score for each frame. The 95% probability is represented as an ellipse.
Herein, we present a case involving real-time intraoperative AI analysis of a patient who underwent laparoscopic anatomical liver resection for colorectal liver metastases located in segment VIII after detailed preoperative planning with three-dimensional simulation (Figure 5). Preoperative three-dimensional (3D) simulation allows for patient-specific preoperative rehearsal. The appropriate liver transection plane is determined, and the anatomical location of intraoperative appearing tubular structures is identified preoperatively. The tumor was in close proximity to the ventral branch of the Glissonean pedicle of segment VIII (G8); hence, laparoscopic anatomical liver resection of segment VIII was planned (Figure 5a). Preoperative patient-specific rehearsal based on 3D simulation was performed to visualize the liver dissection plane, exposing the middle hepatic vein (MHV) and hepatic vein of segment VIII (v8) (Figure 5b). The actual LH surgical monitor was connected to the AI, and real-time analysis of the surgical video was performed at a location that did not affect the surgeon’s field of view. Consequently, MHV, v8, and G8 appeared as expected (Figure 5c), and the AI was able to recognize the structures in real time (Figure 5d).
Preoperative patient-specific simulation and real-time intraoperative AI prediction. (a) Preoperative 3-dimensional simulation image. (b) Enlarged image of the black square in (a), patient-specific simulated rehearsal was performed. (c) An original frame. MHV and Glissonean pedicle are shown. (d) Prediction by AI model. Hepatic vein and Glissonean pedicle are clearly colored and highlighted in blue and green, respectively. MHV: Middle hepatic vein; IVC: inferior vena cava; G: Glissonean pedicle; V: vein, vent: ventral branch; AI: artificial intelligence.
Discussion
This study is a breakthrough and revolutionary world-first attempt to build an AI system that recognizes and color-codes the tubular structure of the visible liver. Intraoperative bleeding and postoperative bile leakage, which are adverse events associated with vascular structures, are the main concerns during liver surgery. Approximately 30% of surgical complications are caused by misidentification during surgery (5). Hence, preventing misidentification of vascular structures is critical in reducing surgical adverse events and complications in liver surgery. The present study demonstrated that the constructed AI system was able to automatically recognize and color-code tubular structures appearing on the liver transection plane. The accuracy and misrecognition rate of the constructed AI system were comparable to those of a skilled HBP surgeon. In particular, the misidentification rate of the constructed AI system was low. As soon as structures appeared during liver dissection, their accurate recognition by the surgeon using AI enabled their proper handling, thereby avoiding inadvertent damage and promoting precise and safer LH and fewer complications.
In recent years, AI using deep learning has begun to be widely adopted in the medical field; e.g., in diagnostics in the fields of radiography (7-9), endoscopy (10, 11), and pathology (12, 13) as well as in the field of surgery to indicate safe areas and anatomical structures for laparoscopic cholecystectomy (14-16). Furthermore, one study presented and sensationalized the use of AI for the recognition of loose connective tissue in gastrectomy, ushering in a new era of AI-based surgical guidance (6).
Our AI model was able to recognize and highlight tubular structures from a large number of image pixels representing organs, anatomical structures, and surgical instruments. Through highly sophisticated annotation and data augmentation, the detailed and fine surface structures and tubular morphology of each hepatic vein and the Glissonean pedicle could be learned and recognized separately as well as color-coded in real-time analysis.
In the present study, the basic annotation strategy and the flow of the objective AI function evaluation method were based on a previously reported study by the co-authors (6). Notably, AI recognition achieved high accuracy. The sophisticated annotation was capable of achieving very accurate tubular structure recognition with high sensitivity (mean 4.24; range=2.8-5.0) and low misrecognition rate (mean 0.12; range=0-0.6; Table I). In Figure 4, the mosaic diagram and the scatter plot visually and statistically demonstrated the relationship between sensitivity scores and misrecognition rates. High sensitivity and misrecognition rate scores were represented.
The AI system accurately highlighted the tubular structures almost as soon as they appeared during liver dissection. The recognition of these tubular structures was found to be as good as that by a skilled and trained HBP surgeon, with these structures colorized and displayed in real time. This time gap is expected to be shortened with improvement of the device. Upon approval as a medical device in near the future, surgeons should not feel much discomfort with the time gap when using this AI system as a submonitor in the operation room.
The AI system’s ability to represent structures in color is truly distinctive. Instead of displaying the tubular structure in a solid color, the AI system recognizes the minute surface features and delicately, sharply, and naturally displays them, allowing the observer to recognize the 3D effect of the tubular structure as well. Moreover, the coloring is natural and nonirritating and does not interfere with other important visual information in the monitor.
ICG fluorescence imaging (ICG-FI) and the present AI system exhibit high affinity, which allows for the clear coexistence of the fluorescent green ICG coloration and AI coloration. ICG-FI provides information on tumor location, margin, and liver segment (3, 17, 18), whereas our AI system provides information on tubular structures related to liver resection. Therefore, more than ever, we believe that by reflecting the effects of both ICG and AI simultaneously, it is possible to achieve the ideal surgery envisioned preoperatively (Figure 3). Another advantage is that the developed AI system can be widely used given its high compatibility with various camera devices from several companies. Although the present study only used 1588 AIM PINPOINT System (Stryker Japan, Tokyo, Japan) data for annotation, deep learning using images from various cameras will be promoted to further improve the accuracy of the system in the future.
For medical students and residents who have not been educated about the liver microstructure and have no prior knowledge of the same, this AI system will allow them to efficiently understand the structure based on the images analyzed by the AI system. As shown in Table II, the usefulness of the AI system for recognition by medical students and residents was rated as very high, scoring 1.8 out of 2. In addition, 5 of the 10 still images received full marks. The AI system is also expected to have a secondary effect of deepening medical students’ and residents’ understanding of anatomy and surgical processes by making them recognize dissections directly and visually, thereby increasing their interest in surgery and surgical science. In light of the above, this AI system is expected to be of great value in medical education.
Given the complex anatomy of the liver and considerable variations among individuals, preoperative knowledge about the anatomy of each individual patient is quite obviously extremely important. In particular, preoperative anatomical knowledge may be more important during laparoscopic surgery than during open surgery considering the lack of tactile sensation, limited field of view, and limited use of ports or forceps in the former. Preoperative 3D simulation allows surgeons to grasp important anatomical landmarks and anatomical relationships and facilitates surgical planning and appropriate liver transection plane determination. As such, the tubular structures appearing on the resection plane intraoperatively can be predicted. This preoperative patient-specific rehearsal has been reported to be useful by our HBP team (19) and is always routinely performed. In LH surgery under detailed surgical planning shown in Figure 5, real-time intraoperative recognition and color-coding of tubular structures using the AI system will improve the surgeon’s vision and enable the entire team to share information. This can further promote surgical progress and surgeon decision-making. However, AI recognition of specific structures has not been realized at present. Nonetheless, we believe that this will be accomplished in the near future through advanced teaching of anatomical location information using GPS-equipped ultrasound equipment or other devices. There is no doubt that automatic recognition of preoperatively predicted targets and landmark structures during surgery through the use of AI would improve the safety and accuracy of surgery.
The eventual and most important aim of automated anatomical segmentation is to assist the surgeon in decision-making. Even with advances in surgical optical instrumentation technology, surgical outcomes depend on the surgeon’s experience and expertise (20, 21) as well as cognitive abilities due to physical and mental conditions (22) during surgery. Moreover, visual support with automatic segmentation technology with AI will compensate for the technical gap recognized between surgeons, thereby improving surgical safety and reducing complication rates. In the near future, we are expecting the medical instrumentation of this AI system.
Despite the clinical relevance of these results, several limitations must be noted. First, the AI system has not yet been trained to accurately identify tubular structures in possible intraoperative situations (e.g., situations in which structures are covered by tissue). Given that the AI system recognizes only those structures whose surface is exposed, it cannot currently be used as a tool that exceeds the surgeon’s experienced prediction. To overcome this challenge, training data on situations in which structures are covered by tissue can be generated from surgical videos of experienced surgeons to improve segmentation performance and enhance functionality. The second limitation is the inability to evaluate AI functions in clinical practice, real-time impact on surgeons, and surgical outcomes due to the unavailability of medical device approval. Considering that medical device approval is forthcoming, the application of this AI system in surgical settings will probably not be far off. And with further environmental improvements and system refinements, the time gap is expected to shorten, and the present AI system can be expected to serve as a useful tool to carry out surgeries more comfortably.
Conclusion
The newly developed AI system is a revolutionary and novel tool that color-codes vascular structures during LH, which strongly relies on visual information.
By visually assisting surgeons in recognizing tubular structures, the AI system enables the surgical team to have a common recognition of the anatomy while addressing the disparity in surgeons’ recognition ability. The AI system can also recognize the hepatic vein and Glissonean pedicle that are suddenly encountered in the liver parenchyma on the liver transection plane, thereby avoiding inadvertent tubular structure injury, enabling appropriate surgical manipulation, and improving surgical outcomes.
Combined with other imaging technologies, such as ICG-FI, preoperative 3D simulation, and GPS-enabled ultrasound, we believe that color-coded LH, which displays tubular structures, liver segments, and tumors in different colors in real time, will soon become a reality.
Footnotes
Authors’ Contributions
KT, TA, NK and YT contributed to the design and implementation of the research. KT, NK, YT, YK and TA contributed to the analysis the data and the development of the AI system. KT, TA, YT, KY and MW contributed to create figures and tables. KT, YT, HS, TH, TY, KS, KM, TK, AF, YE and TA contributed to the operation of the laparoscopic hepatectomy.
Conflicts of Interest
This research did not receive any specific grant from funding agencies in the public, commercial, or not-for-profit sectors. The Authors have no conflicts of interest in regard to this study.
- Received August 8, 2023.
- Revision received August 21, 2023.
- Accepted September 27, 2023.
- Copyright © 2023 International Institute of Anticancer Research (Dr. George J. Delinasios), All rights reserved.
