Project TeleSTAR: AR-based 3D-Live Stream of surgery

First live transmission of surgical insertion of a cochlea implant using an augmented reality-based 3D live stream.

In today’s surgical training, the transmission of surgeries for physicians and students is limited in terms of collaboration and perceived experience. Therefore a team of experts of TU Delft, Charité – Universitätsmedizin Berlin, Fraunhofer HHI and ARRI Medical have come together to solve the problem with the help of Augmented Reality (AR).

The result is TeleSTAR – Telepresence for Surgical Assistance and Training using Augmented Reality. The TeleSTAR system uses a full digital surgical microscope and computer vision algorithms to develop new AR-based training and teaching methods and tools, leading to a new level of educational transparency.

The system offers new ways to visualize relevant surgical information, allowing surgeons to improve, and also explain their decisions, as it guarantees the exact same surgeon’s field-of-view for all attending healthcare professionals. “Simple” standard procedures, as well as rare and complicated procedures, can be visually annotated and augmented – and supplemented with audio comments – to make every step of the surgery transparent. A bidirectional communication platform allows all attendees to address questions to the surgeon at the right time, which creates a dynamic teaching environment despite the local distance. The project involves 1-2 day sessions where ENT procedures will be performed using the “ARRISCOPE”, a digital surgical microscope. ENT physicians from Germany and the Netherlands will attend surgeries, which will be broadcasted to remote lecture rooms for teaching purposes.

On 26 September 2020, the first procedure took place at the Charité Berlin and was transmitted to remote lecture rooms in Delft, where 35 participants from TU Delft, including ENT physicians and master students of the faculty “Biomedical Engineering”, were able to follow the surgery live for teaching purposes. Importantly, the 3D live stream was supplemented by simultaneous audio commentary and an intraoperative annotation mode using AR. The intraoperative annotation mode allowed surgeons to add additional visual information such as sketches, references and image-based anatomy measurements to the live stream. In this way, every step of the surgery was explained and comprehensible.

“Students and clinical trainees must know the full context in teaching, in the modern Medical technology is used. TeleSTAR offers surgical trainees and developers in medical technology precisely this important contextual awareness,” adds Prof. Dr. John vd Dobbelsteen from the Faculty of Biomedical Engineering at the TU Delft.

“Augmented Reality using AI and computer vision algorithms in combination with digital processing chains are the key technologies for new intraoperative assistance instruments and the promotion of training through a new level of surgical transparency,” explains Dipl.-Inf. Jean-Claude Rosenthal, research associate at Fraunhofer HHI.

The next 3D/AR live operation will be broadcast to multiple locations and is scheduled for 14/15 November 2020. More information is available on the official website.