15056 – Smartphone and Tablet Based Augmented-Reality System for Image-Guided Interventions

Augmented Reality (AR) system that overlays diagnostic imaging (e.g CT, MRI, PET, SPECT) on surgical site using mobile devices


Surgical interventions present the challenge to the surgeon of precisely excising cancerous tissue while avoiding damage to healthy tissue and critical structures. Image-guidance systems can be useful in such situations. Typical image guidance uses x-ray (fluoroscopic) or ultrasound imaging with a monitor positioned near the patient. Tracking devices are not normally integrated into such systems, resulting in uncertainty of the position of surgical tools relative to the anatomy.

The novel augmented reality (AR) system integrates a smartphone (or tablet) with navigation/visualization software to overlay the surgical site with diagnostic imaging (e.g. CT, MRI, PET, SPECT). The operating surgeon could then visualize medical images from the smartphone display overlaid on video images captured from the back of the smartphone in front of the patient. Real-time tracking is integrated into this system to track the smartphone location in 3D space so that the medical image can be displayed in the correct manner relative to the subject. In the radiation therapy applications, the prescribed dose of the treatment site can be overlaid on the camera images.

The hardware component of the AR system includes a base computer, optical tracker, and mobile device (Android OS based). The software component includes GTxEyes installed on base computer, and two android applications which are IP camera and virtual network computing (VNC) installed in mobile device. GTxEyes is an in-house software environment used for real-time tracking, visualization, and navigation in image-guided surgery. This software also provides augmented reality by overlaying computer generated images on real patient video image.

Using tracking technology, the mobile device is localized in real-time and 3D space with affixed tracking tool. The IP camera application on mobile device continuously streaming video images to the base computer via wireless network. The GTxEyes software processes the video and overlay medical image and/or computer generated virtual objects (such as lesions and critical structures). Simultaneously, the VNC application on the mobile device allows end users to visualize the augmented video on the base computer from the mobile device display via wireless data transmission. The communication process also works through local wireless networks (both base computer and mobile device need to be on the same wireless network). To ensure security of video streaming, both android applications IP camera and VNC are executed with security password login. No patient information is required to be stored in the mobile device.

Potential Applications

The smartphone (or tablet) based AR system has potential across a spectrum of image-guided procedures including surgery, interventional radiology, and radiation therapy. The integration of real-time tracking with AR system would allow the smartphone/tablet to move freely in 3D space and provide a variety of perspective views of subjects while maintaining system accuracy without re-calibration.

· In head and neck reconstructive surgery the surgeon can harvest a free-flap by guidance of the AR device overlaying the anatomical structures (e.g. perforator, arteries, vein).

· For lesion ablation, surgeons can delineate the critical structures and lesion from the medical images (e.g. CT, MR, PET, SPECT). Such delineated subjects can be overlaidon the camera images that are acquired from the mobile smartphone/tablet device. These features can be used for pre-intervention planning in the clinic, intraoperative guidance, and post-operative follow up.

· Similarly, in radiation therapy, the prescribed dose of the clinical target including CTV, GTV and HTV can overlay on the treatment site.

Contact Information

Name: Noah Schwartz

Email: Noah.Schwartz@uhnresearch.ca

Phone: 0000000000