Segmentation of subretinal and intraretinal fluid on OCT B-scans using an easily fine-tunable and deployable, visual-prompting based computer vision model

Introduction: Segmentation tasks, particularly for OCT images, typically require significant time investments and rely heavily on expert annotators. Also, currently available segmentation tools have limited adaptability or customization to new tasks. Here we present a novel computing tool based on visual prompting. This tool facilitates the fine-tuning of segmentation algorithms based on the segment anything foundation model (SAM, Meta AI), requiring minimal training input while delivering satisfactory performance even with few shot experiments. Methods: We employed a custom-made visual prompting-based computing tool to streamline the segmentation process based on the SAM algorithm (IBM Research, Zurich). The tool was tested on the segmentation of subretinal and intraretinal fluid (SRF and IRF, respectively). 125 images of the openly accessible Retinal OCT Image Classification C8 dataset (Kaggle) classified with diabetic macular edema were manually segmented by an expert for SRF and IRF with the open-source tool 3D Slicer. For the validation of the tool, 100 images were used. For performance evaluation we calculated the intersection over union (IoU) between ground-truth and predicted segmentations. We performed experiments of 1 shot and 5 shots using either 1 or 5 images as reference respectively, and 20 as training dataset.
Results: The 1-shot experiment, finetuned on 20 images, yielded an average IoU of 0.191. When the number of reference images was increased to 5, the average IoU also increased to 0.332 (average IoU of 0.342 for SRF and 0.321 for IRF). Conclusion: The implementation of the visual prompting tool yielded promising results, showcasing its potential to significantly aid in segmentation tasks of IRF and SRF in OCT images. This study highlights the effectiveness of a visual prompting-based tool, providing a user-friendly, efficient, and flexible approach to segmentation tasks in OCT imaging. Furthermore, only very-light supervised fine-tuning has been used in our tests which contrasts with extensive fine-tuning necessary for the current state-of-the-art segmentation tools. While the results of our pilot study are below state-of-the-art segmentation metrics, further developments on algorithm fine-tunning for still minimal manual segmentation input, will likely enhance the potential of our proposed framework.