We propose a three-dimensional (3D) multimodal medical imaging system that combines freehand ultrasound and structured light 3D reconstruction in a single coordinate system without requiring registration. To the best of our knowledge, these techniques have not been combined as a multimodal imaging technique. The system complements the internal 3D information acquired with ultrasound with the external surface measured with the structured light technique. Moreover, the ultrasound probe's optical tracking for pose estimation was implemented based on a convolutional neural network. Experimental results show the system's high accuracy and reproducibility, as well as its potential for preoperative and intraoperative applications. The experimental multimodal error, or the distance from two surfaces obtained with different modalities, was 0.12 mm. The code is available in a Github repository.
Overview
Our proposed multimodal system is shown in Fig. 1 which consists of two cameras, a DLP projector, and a B-mode ultrasound (US) machine. The figure shows the main coordinate frames involved in the multimodal imaging technique.
Figure 1. Multimodal 3D imaging system composed of two cameras, a projector and a US machine.
The main idea is to obtain the external surface of the patient or zone of interest with the structure light technique using \( \{Cam_1\} \) and the projector \( \{P\} \), and the internal structure with the 3D freehand ultrasound method by mapping the ultrasound slides \( \{I\} \) to the 3D space using \( \{Cam_1\} \) and \( \{Cam_2\} \) to track the pose of a target \( \{T\} \) attached to the ultrasound transducer. 3D data acquired from both modalities are referred to \( \{Cam_1\} \) coordinate system which is also the world \( \{W\} \) frame for structure light and freehand ultrasound techniques.
The reconstruction pipeline is described in Fig. 2, where in a first stage fringe patterns are projected onto the region of interest to obtain the 3D surface. In a second stage, we acquire US images from the internal structure of interest for 3D reconstruction with freehand ultrasound. For this technique we use the stereo vision system \( \{Cam_1\} \) and \( \{Cam_2\} \), a marker of three circles, and MarkerPose: a sub-pixel center detection method based on deep learning for pose estimation of the ultrasound probe. Then, in this second stage, as input we have the stereo images of the marker, and an ultrasound slice which is mapped to the 3D space.
Figure 2. The proposed multimodal 3D imaging pipeline with the external 3D surface and internal US images.
Breast Phantom Results
For multimodal evaluation, we used a breast phantom 3B SONOtrain P125 consisting of three tumors:
Figure 3. Breast phantom model used for multimodal reconstruction.
The experimental setup for this experiment is shown in the following image:
Figure 4. Experimental setup for the breast phantom experiment.
The acquisition process for external and internal reconstruction is shown below:
Video 1. Acquisition stages for multimodal reconstruction.
which leads to the following multimodal results (set the opacity to 1 to see the texture of the external surface):
Concentric Cylinders Results
We also built an experiment consisting of two concentric cylinders, where the inner cylinder is submerged in water to be measured with freehand ultrasound.
Figure 5. Measured object: outer and inner cylinders.
For the first stage we project fringe patterns:
Video 2. Fringe projection onto the external cylinder.
For the second stage, the ultrasound images are acquired and the pose of the marker is estimated:
Video 3. Ultrasound slices acquisition of the internal tumor.
The multimodal results of the internal and external cylindrical objects are shown below:
Potential Applications: Navigation System
As we can obtain the pose of the marker in real-time, we can use this for medical navigation by displaying the ultrasound images and the 3D model of a transducer in a 3D real-time visualizer.
Video 4. Multimodal technique as a navigation system.
For navigation purposes, we developed a real-time visualization software using OpenCV 3D Visualizer, where we merge MarkerPose with the stereo visualization of the cameras.
Video 5. Example of the navigation visualization system.
Potential Applications: Projection-based Guidance
In addition to using the projector for fringe projection, we can exploit it to point out the location of the tumors.
Figure 6. Using the projector to highlight the location of the tumors.
We can also project the reconstructed tumors with the ultrasound slices onto the phantom surface including their texture. This as a first step towards projection-based augmented reality guidance. Note that the projected tumors are in agreement with the actual position and texture of the tumors.
Figure 7. Towards projection-based augmented reality applications.
Citation
@article{meza2021three,
title={Three-dimensional multimodal medical imaging system based on freehand ultrasound and structured light},
author={Meza, Jhacson and Contreras-Ortiz, Sonia H and Romero, Lenny A and Marrugo, Andres G},
journal={Optical Engineering},
volume={60},
number={5},
pages={054106},
year={2021},
publisher={International Society for Optics and Photonics}
}
Acknowledgments
This work has been partly funded by Universidad Tecnológica de Bolívar project C2018P005. Jhacson Meza thanks Universidad Tecnológica de Bolívar for a post-graduate scholarship and MinCiencias, and MinSalud for a “Joven Talento” scholarship.
