A CONCEPT WORK FOR AUGMENTED REALITY VISUALISATION BASED ON A
MEDICAL APPLICATION IN LIVER SURGERY
Tim Suthau! , Marcus Vetter‘, Peter Hassenpflug”, Hans-Peter Meinzer”, Olaf Hellwich'
'Technical University Berlin, Photogrammetry and Cartography
EB 9, StraBe des 17. Juni 135, 10623 Berlin, Germany
Deutsches Krebsforschungszentrum, Div. Medical and Biological Informatics
Im Neuenheimer Feld 280, 69120 Heidelberg, Germany
Commission V, WG V/3
KEYWORDS: Augmented Reality, Medicine, Photogrammetry, Visualisation, Close Range
ABSTRACT:
This paper describes a concept of an innovative medical application with augmented-reality support. This work is based on a
cooperation between the Div. Medical and Biological Informatics at Deutsches Krebsforschungszentrum (DKFZ), and the
Department of Photogrammetry and Cartography of the Technical University Berlin.
Augmented Reality (AR) is a growing area in virtual reality research. An augmented reality system generates a composite view for
the user. It is a combination of the real scene viewed by the user and a virtual scene generated by the computer that augments the
scene with additional information. The wide scope of application domains reveals that the augmentation can assume a number of
different forms. Therefore, we compare different technologies for augmented reality visualisation with the focus on video see-
through head-mounted displays, optical see-through head-mounted displays and virtual retinal displays. In a further step, we analyse
the state-of-the-art of medical applications in augmented reality visualisation.
New technologies and methods implemented in DKFZ’s ARION™ software will enable image-guided liver surgery. It is shown that
many tools for preoperative planning of surgical interventions in hepatobiliary and pancreatic (HBP) surgery are available in the
clinical environment, while the surgical procedures itself still lack computer assistance. On this basis, Div. Medical and Biological
Informatics at DKFZ, the Depts. of Surgery and Radiology at the University Clinics Heidelberg and the research group “AR-work”
of Technical University Berlin derived strategies for AR-support of open liver surgery. A technique will be developed in order to
superimpose virtual computer-generated information with real patient image data using a see-through technology. The data from the
preoperative planning, the clinical and technical requirements for augmented-reality techniques and our concept of the augmented-
reality system will be presented in this contribution. The advantages of see-trough technology will be discussed. Our future work will
analyse why (nowadays) AR-systems are not available in surgery and which technical and clinical requirements can be fulfilled with
currently available display technologies.
1. INTRODUCTION
Because imaging technology is so pervasive throughout the
medical field, it is not surprising that this domain is viewed as
one of the more important for augmented reality systems. Most
of the medical applications deal with image guided surgery. Pre-
operative imaging studies of the patient, such as CT or MRI
scans, provide the surgeon with the necessary view of the
internal anatomy. From these images surgery is planned.
Visualisation of the path through the anatomy to the affected
area, where, for example, a tumor must be removed is done by
first creating a 3D model from multiple views and slices in the
preoperative study. This is most often done mentally though
some systems create 3D volume visualisations from the image
study. Augmented reality can be applied so that the surgical
team can see the CT or MRI data correctly registered on the
patient in the operating theatre while the procedure is
progressing.
AR has some very interesting applications in medicine but most
of them are still in the prototype stage. The most important
applications will be summarised in the paper.
Our concept relies on the research work of the Div. Medical and
Biological Informatics at DKFZ Heidelberg in surgery planning
for liver resection:
The planning of surgical interventions of the large inner organs
like the liver can be based on a computer supported planning
system. The complex anatomical structure of the liver including
two venous vessel trees, one arterial vessel tree and the bile duct
is hard to understand without three-dimensional reconstruction.
Additionally, anatomical information is important during
surgery concerning the position of the eight liver segments and
in particular the position of their interfaces. The anatomical
variation between different patients demands individual
operation planning in this field. A software system including the
whole planning procedure is developed and is embedded in the
radiological workstation CHILI®. The planning procedure
starts with the segmentation of important areas (liver, tumor,
etc.), the processing of the vessel trees, and it ends with the
proposal of an individual surgical strategy. The next mandatory
step is the integration into the operation theatre. Here
intraoperative visualisation and navigation of the surgical
instruments will be used.
Nowadays, augmented reality (AR) allows for enhanced
perception of the surgical situs by superimposing stereoscopic
projections over the field of operation. But the most AR-
applications are used for the preoperative planning and not for
the intervention itself. Our future work aims at elaborating the
best display-technology for liver surgery (Herfarth, 1998;
Hassenpflug, 2001).
-274—
FS. 5 Rud ml
ub Peau bed TO C jon. MN je
