Mixed-Reality Headset Procedural Targeting

The Mixed-Reality Headset Procedural Targeting system was created for procedural targeting assistance and visualization for medical students, residents, fellows, and other healthcare providers. Using Microsoft HoloLens 2 and Magic Leap 2 mixed-reality headsets, a system was developed to spatially designate targets inside of a simulated patient for enhanced visualization and needling guidance. In addition to the headsets, a Surface Pro laptop and a simulated patient, ultrasound probe, and needle were used.

Using the mixed-reality headsets, the simulated patient and ultrasound probe are spatially tracked using QR codes. The ultrasound probe is used to find a target inside the patient. By touching the target on the ultrasound screen, a holographic sphere is created and placed in the 3D space inside the patient. This target is viewable at any angle inside the mixed-reality headsets. The probe can then be removed from the simulated patient and the target will remain at the proper position. The needle is then driven to the target in the simulated patient using the holographic sphere and depth markers on the needle. To validate the system, electromagnetic tracking of the needle tip was used, independent of the headset tracking, for precise determination of the distance from the needle tip to the center of the target. The video below demonstrates the system.

CSSALT worked with colleagues from the UF Department of Anesthesiology to develop this system.