Hand Tracking with Computer Vision and Spatial Co-Registration in a Mixed Reality Procedurally Simulator
Webpage created 9/29/2025
Peripheral intravenous (IV) placement is among the most frequently performed procedures worldwide. We built a mixed-reality trainer that uses depth cameras and computer vision to track the learner’s hands, co-registering them in 3D with an electromagnetically tracked needle, ultrasound probe, and simulated anatomy. This unified spatial view lets the system recognize procedural steps and provide step-by-step, real-time coaching, enabling consistent practice without requiring an in-room instructor.
