Mixed reality is one of the newest and fastest growing fields of technology and University of Nebraska-Lincoln researchers are on the forefront of this new frontier.
Mixed reality is the merging of real and virtual worlds to produce new environments and visualizations where physical and digital objects co-exist and interact in real time.Trevor Fellbaum, a junior computer science major, is part of the university’s Mechanical and Material Engineering Department within the College of Engineering. The department accommodates a multitude of students working inside the Nebraska Hall Lab.
Fellbaum’s project is focused on the development of a miniature robot that can perform surgical tasks. His specific undertaking in the project is to create a virtual holographic controller within the HoloLens, a pair of mixed reality smartglasses released by Microsoft in March 2016, then also integrate that with the existing technology used with the robot.
One of the problems that researchers faced is that there used to be a physical controller which required a surgeon to be close to the action to control the robot. With the HoloLens and the virtual holographic controller, the surgeon will be able to control the robot long-distance using a wireless internet connection.
The robot would primarily be used for helping serve our military overseas. A doctor would be able to provide preliminary care to a wounded soldier without ever being in danger themselves.
“I think that it is an important project that a lot of people would find interesting, especially with its ability to save lives,” Fellbaum said.
The HoloLens has many other applications, including helping doctors practice surgeries with simulations and assisting educators deliver remote instruction to others who have the device.
One of the major limitations the engineers face in their research is the HoloLens only has centimeter tracking, while many precise surgical procedures require at least millimeter tracking.
In spite of the difficulties, progress is still being made. The first human test of this robot was completed within the last year, as the robot was successfully used in a colon resection procedure in Paraguay.
Fellbaum said he expects that it will be somewhere between eight to 10 years before the robot is able to be fully implemented in the battlefield.
The union of the older technologies of the robot with the newer technologies of the HoloLens has ushered in new innovations and possibilities to the researchers at the university.
Fellbaum recently joined the team, but the project as a whole has been in development for the past decade by professor Shane Farritor and assistant professor Benjamin Terry.
Iman Salafian graduated from the university with a Master of Science but still works as a part-time researcher and is the lab manager for the Terry Research Laboratory.
“[Terry] is a super nice person,” Salafian said. “He spends the majority of his time trying to advance progress on his research, and he tries to involve students in most aspects of the research and development process.”
While headway is being made, Fellbaum is still focused on the work ahead.
“The next steps in our research is to add visual streaming to the HoloLens from the robot so you can see what the robot is seeing,” he said.