A Miniature Multimodal Imaging System for Medical Diagnostics and Interventions in Space

The overall goal of the project was to develop a prototype miniature multimodal imaging system for guiding medical examinations and interventions during space exploration. We adapted the wearable imaging goggle concept for aerospace use and expanded on the present surgical goggle system by integrating multiple imaging modalities, including pre-flight MRI/CT, microscopy and real-time ultrasound, with the fluorescence goggle. The 2D/3D multimodal images were processed using computer vision algorithms, co-registered and displayed in the augmented reality goggle eyepieces in real-time. The goggle system was designed to be compact, portable and user-friendly, facilitating its use in space by astronauts. The project was guided by three specific aims: (1) Develop and characterize a prototype goggle device that can offer multimodal imaging, spatial tracking and wireless communication capacity, (2) Integrate ultrasound, microscopy, infrared imaging and MRI/CT with fluorescence images and display on goggle eyepieces, (3) Use the multimodal imaging goggle to guide medical examinations and interventions by first responders, and implement a ultrasound guidance and virtual training program. APPROACH Optical imaging for fluorescence guided surgery has been reliant on bulky equipment (i.e. Novadaq Spy, FLARE, Perkin Elmer Solaris). While these systems can supply sensitive fluorescence detection and fine optical imaging resolution, their size constraints make them inadequate for expedition, in-transit or aerospace imaging. The recent surge in virtual and augmented reality systems (e.g. Micorsoft Hololens, Meta 1, Vuzix) have provided an avenue for relief of the size constraint, while giving opportunity for additional advances in the quality of optical imaging and augmented reality display. Our project builds on the foundations of wearable medical imaging systems established by prior studies in the sub-fields of fluorescence and ultrasound imaging [1-4], as well as biomechanical visualization [5]. Initially, we advanced the field by integrating stereoscopic fluorescence imaging sensors onto a wearable augmented reality (AR) display. Imaging sensors were placed in-line with the AR display screens, which were positioned in front of the user’s eyes, enabling line-of-sight imaging, Figure 1A. Stereoscopic color imaging sensors were also incorporated, Figure 1B, into the system design, allowing for color to fluorescence image co-registration. The system differs from previous models reported in the literature [6-8] by implementing an entirely software based, marker-less method for stereoscopic co-registration. The goggle cameras were characterized for fluorescence detection sensitivity and resolution, as well as color to fluorescence image registration accuracy. The compact design allowed for use in a variety of environments where size, weight and portability are a factor. Figure 1 A. Wearable fluorescence imaging goggle incorporating stereoscopic imaging sensors and near-eye display, affixed to a loupe-style head mount. B. Modified goggle incorporating both near-infrared (NIR) fluorescence sensors and color imaging cameras. Additional progress was made on the fluorescence imaging module through the incorporation of a pulsed light fluorescence excitation regime. Prior systems have implemented various forms of pulsed excitation and gated image acquisition for fluorescence imaging [9-11]. However, these systems use fixed, stand-mounted imaging and display modules. Our system, being wearable and mobile, was subject to motion artifacts during gated acquisition. We developed a fluorescence identification algorithm for use during pulsed excitation imaging which greatly reduces the presence of both motion artifacts and other false positive fluorescent labels. Next, we further progressed research by co-registering real-time ultrasound and pre-procedural 2D/3D CT/MRI imaging data with the live fluorescence imaging data, and displaying the results to the user via the head-mounted AR display. Ultrasound imaging data was initially displayed along with the fluorescence imaging data in picture-in-picture mode, and later co-registered to the fluorescence image via optically tracked fiducial markers, Figure 2A-B [12]. Pre-procedural MRI data was compiled into a false colored 3D object, which was registered onto the scene with similar fiducial markers, Figure 2C. The field was advanced through the incorporation of a method for 3D registration correction. The technique used an optically tracked stylus to allow the user to mark and correct registration errors. Both registration techniques were analyzed for fiducial marker registration error (FRE) and target registration error (TRE). Figure 2 A. Goggle view incorporating fluorescence imaging frames with ultrasound, shown top right. B. Fluorescence imaging (green) frame co-registered with ultrasound image (purple). The ultrasound image was registered to the optically tracked fiducial markers affixed to the transducer. C. Reconstructed MRI object (heart) co-registered via fiducial markers to a tissue phantom containing fluorescent markers (blue arrows). In addition to ultrasound and MRI/CT data incorporation, we developed and implemented a near-infrared vein imaging module. Unlike prior works (e.g. VeinViewer, AccuVein) which utilize reflectance mode imaging and a monochrome projected light display, we implemented a transmission mode regime for vein imaging with reflectance mode for color anatomical imaging, Figure 3. The two modes were combined via beam splitter, providing an output display image containing both anatomical and physiological (i.e. vein) imaging data. Figure 3 A. Schematic of beam splitter device for obtaining infrared and color imaging data. B. The device in use. Multi-scale imaging was implemented by integrating a fiber optic microscopy module into the imaging system. The microscope design was based on prior work [10], but advanced the field by integrating the microscopic module with the wide field-of-view imaging provided by the goggles. Microscope images were displayed along with goggle images in picture-in-picture mode, Figure 4. The microscope was characterized for fluorescence detection sensitivity and resolution. Figure 4 A. Schematic of fiber optic microscope. B. Image of fiber optic microscope. C. Image of resolution test target obtained using the microscope with a 20X objective. Minimum resolvable dimension was found to be about 7µm. D. Goggle image combining fluorescence (green) with a microscopic image on the same target (top left). Decision support algorithms were developed for the integrated ultrasound imaging module, leveraging on convolutional neural network (CNN) architectures. Prior works have used CNNs for ultrasound image identification and object detection [13-16]. Our efforts were different in that we applied a two-tier CNN classification system, the first tier being used to identify the entire ultrasound image while the second tier localized target objects within the image. Additionally, we developed a Histogram of Oriented Gradients Surface Vector Machine (HOG/SVM) classifier for the identification of OCT retina images. Lastly, we translated the technology to multiple applications. We worked with clinicians and researches at the Cleveland Clinic to conduct multiple fluorescence imaging studies on pigs, including perfusion assessment and wound healing monitoring. Additionally, we have initiated multiple human clinical trials for fluorescence guided tumor resections and pre-operative planning. Additionally, we have translated the system to forensic imaging for the detection of latent blood stains and chemical targets of interest. RESULTS Characterization of the imaging sensors revealed minimum fluorescence detection limits of below 10 nM concentration for three tested clinically relevant fluorescent dyes (indocyanine green (ICG), fluorescein, and protoporphyrin IX (PpIX)) in a dark room setting. Additional tests using tissue phantoms were conducted, and indicated minimum fluorescence detection of about 100 nM for ICG and 150 nM for PpIX. Additionally testing was conducted using not only various dye concentrations, but also various fluorescence excitation intensities, dye volumes, imaging working distances and tissue phantom imaging depths. Results were tabulated and plotted to provide a wider report of fluorescence characterization than the single minimum detection concentration, which is typically reported in the literature, Figure 5A. We also demonstrated improved fluorescence detection with the use of pulsed excitation with gated image acquisition over traditional steady state excitation, Figure 5B. Figure 5 A. Plot of the dye concentration required to achieve a signal to background ratio of 2, as imaged by the goggle cameras. Readings were taken while varying the excitation intensity, dye volume and tissue depth of fluorophore. B. SBR versus dye concentration plot compares pulsed light and steady state imaging, and demonstrates consistently improved detection for pulsed light. A method for quantifying color to fluorescence image co-registration error was developed using a chessboard pattern. Error was averaged in the xy-plane of the co-registered images at values of less than 1 mm in either direction, Table 1. Ultrasound and MRI image registration was conducted to the fluorescence imaging frames using optically tracked fiducial markers. Registration was conducted at video rate (> 15 fps), with FRE and TRE recorded between 1-2 mm in the x and y directions on the imaging plane, Table 2-3. Table 1 Error measurements for the fluorescence to color registration algorithm. Measurements included translation error in the x-axis (Tx) and in the y-axis (Ty), as well as rotation error (R) and scaling error (S). Tx (mm) Ty (mm) R (o) S (%) Mean 0.73 0.96 1.66 1.65 STD 0.50 0.56 0.91 0.02 Max 2.0 1.9 4 5 Min 0.1 0.1 0.3 0 Table 2 Error measurements for ultrasound to fluorescence image registration using Aruco fiducial markers. Measurements included translational error in the x-axis (Tx) and in the y-axis (Ty), as well as rotational error (R) and scaling error (S). Both the Fiducial Registration Error (FRE) and Target Registration Error (TRE) were determined. FRE TRE Tx (mm) Ty (mm) Tx (mm) Ty (mm) R (o) S (%) Mean 1.22 1.15 1.86 1.81 2.19 3.89 STD 0.35 0.30 0.53 0.55 1.42 2.07 Max 1.8 1.8 2.50 2.55 6.00 7.50 Min 0.7 0.8 0.80 0.80 0.50 1.10 Table 3 Error measurements for 3D object registration using Aruco fiducial markers. Measurements included translational error in the x-axis (Tx) and in the y-axis (Ty), as well as rotational error (R) and scaling error (S). Both the Fiducial Registration Error (FRE) and Target Registration Error (TRE) were determined. FRE TRE Tx (mm) Ty (mm) Tx (mm) Ty (mm) R (o) S (%) Mean 1.41 1.11 1.74 1.37 0.95 1.01 STD 0.31 0.37 0.47 0.36 0.63 0.00 Max 1.9 1.8 2.4 2 2.5 1.8 Min 0.8 0.5 0.8 0.8 0.1 0.4 The vein imaging module was tested for vein identification in comparison to traditional visual inspection, Table 4. Results indicated an approximate 2-fold increase in the number of veins identified when using the system. Imaging was conducted on both dominant and non-dominant hands, and a difference was found between the mean vein counts. Two distinct computer vision algorithms were implemented in processing the co-registered infrared and color images, Figure 6. Table 4 Vein counts by visual inspection and using the beam splitter imaging system for our 25 test subjects. Improvement Factor was defined as the ratio between system and visual counts. The p values from paired t-tests between visual and assisted vein counts indicated that the means were statistically different for both dominant and non-dominant hands. Hand Counts Mean STD Improvement Factor P - Value Visual System Visual System Visual System Dominant 183 375 8 16.3 2.4 4.6 2.0 1.73E-10 Non-Dominant 211 341 9.2 14.8 3.0 3.5 1.6 2.84E-11 Figure 6 Images acquired using the beam splitter device. A. Color reflectance image, B. Infrared vein image, C. first method of image processing for combining the color of infrared images, and D. second method for image processing. Ultrasound image classification accuracy using the CNN classifier was assessed both on the training and test imaging data sets. The CNN object detector has been deployed, but is still undergoing optimization for accuracy. The two-tier method has been demonstrated for feasibility in real-time classification, Figure 7. Figure 7 A. Input ultrasound image into the classification regime. B. Output image from the classifier, identifying the anatomical location from which the image was recorded and localizing veins within the image. While clinical studies are still ongoing, some early results have proven promising, Figure 8A. Additional animal studies on wound healing, Figure 8B, have demonstrated the ability to track wound progression through visualization of bacterial fluorescence in the wound. Perfusion in a pig leg was assessed via IV angiography, Figure 8C. Lastly, the goggle system was translated for forensic imaging application for the detection of latent blood stains and tested on a variety of surfaces, such as carpet, fabric, wood, glass and tile. Detection sensitivity was observed for blood:water dilutions down to a 1:100,000 ratio on certain surfaces. Figure 8 Applications of the imaging goggle system, including A-B. Fluorescence and reflectance mode image of burn wounds on a pig back, C. fluorescence angiography of a pig leg, and D. clinical study on sentinel lymph node mapping.