Augmented Reality In the Operating Room (ARIOR)

To avoid postoperative complications at brain tumour surgery, the surgeon must visualise both the tumour and functional brain centres for, e.g., speech and motion. This project, representing image-guided surgery, aims at using augmented reality techniques for presenting preoperative Magnetic resonance imaging (MRI) data merged with the optical view of the patient. MRI images will be presented in the operating room with one or more of the following techniques: 1. Operating microscope 2. Video see-through micro-display 3. Hand-held devices 4. Projector-based augmented reality. The augmented reality image will contain MRI information superimposed on the optical image and will be deformed in real time as the brain is deformed during operation, using video angiography or ultrasound to follow the brain deformation. The demonstrator project, to be performed in collaboration by LiU’s Center for Medical Image Science and Visualization (CMIV), XMReality Research AB and clinically active neurosurgeons as well as experts in human-system interaction, comprises construction of a tool for this purpose and evaluation by laboratory tests where neurosurgeons will attempt to reach predefined anatomical targets with and without access to the novel technique. If successful, this project has the potential to reduce the rate of tumor recurrence as well as the risk of post-operative paresis or aphasia, thus reducing human suffering and society’s costs for medical care.

• Staff:

Torbjörn Gustafsson , M. Sc.		research engineer		XMReality Research AB
Per Carleberg , M. Sc.		research engineer		XMReality Research AB
Filipe Marreiros , M. Sc.		doctoral student		Radiology (IMH) / Visualization (ITN)
Susanna Nilsson , Lic. Sc.				Computer Science
Örjan Smedby , Prof.		Principal Investigator		Radiology IMH/LiU, DC/LiÖ

• Former Staff:

• Project Description:
AIM To create an augmented reality tool for safer and more efficient image-guided brain surgery.


WORK PLAN WP1: Hardware solutions The following hardware solutions will be considered and one or more of them will be implemented: 1. Stereoscopic display integrated in an operating microscope.  2. Video-see-through micro-displays in front of each eye offering stereoscopic vision.  3. An "electronic magnifying glass" (hand-held device), possibly consisting of a mobile phone with camera and display.  4. Projector-based augmented reality utilising a miniature projector integrated with a tracking system and possibly a laser radar scanner and placed on the forehead of the surgeon.

 WP2: Software solutions for image fusion  The system will make use of a combination of depth cues: motion parallax (motion of the observer relative to the patient); surface reflection; occlusion of distant objects by closer, more or less opaque, objects; and stereoscopy. Tracking, the problem of estimating the position of a moving observer relative to a target, can be solved in several ways within our project. Our initial plan is to use a combination of optical and inertial trackers. With this laboratory setup, we will be able to test, measure and verify the functionality of our system. Tracker interfacing will be implemented using software developed within our group.

 WP3: Compensating for brain deformation A recently developed technique for studying the surface of the brain in real-time utilizes fluorescence for visualizing the vessels on the brain surface, which are also visible with MRI. Our main plan is to use this technique for solving the brain shift problem.

WP4: Demonstrator setup and evaluation As a system for integration in the clinical routine is not likely to be available within the project time frame, testing by prospective users cannot take place in the operating room. We will therefore implement a demonstrator set-up with the visualization and interaction equipment and access to processed preoperative images and ICG video angiography sequences or 3D ultrasound recordings acquired in clinical routine.


IMPORTANCE In neurosurgery, accurate real-time intraoperative visualization of correctly updated images of the lesion to be removed and structures that need to be protected can represent the difference between cure and relapse or between a healthy life and severe disability after the operation. If this project is successful, the new augmented reality presentation will increase the possibilities for effective surgical treatment of brain tumours.