Tech Allows ROV Pilots to ‘Feel’ the Sea

ROV pilots could soon feel the same currents, rough seas and shuddering as the underwater robots they guide beneath the waves for inspections, installations or repair operations.

New haptic technology — think video game controllers that vibrate during a smackdown — provides a range of tactile sensations for remote pilots that seemingly stops just short of seasickness.

Typically, ROV operators control their underwater vessels from a chair in a tech-heavy control room onboard a vessel or in an onshore operations center far from the water. The pilot watches screens and live video feeds and uses a joystick to maneuver the ROV.

But new tech could find its way into the control room—and onto the body of the ROV pilots in the form of haptic suits. Haptic technology allows the user to perceive — through force feedback, motion and vibration — the murky environments they steer their underwater vessels through. The goal of the technology is to improve control of the vessels.

Through a project funded through the National Science Foundation’s “Future of Work at the Human-Technology Frontier” program, ABS and the University of Florida are pursuing the development and qualification of the augmented sensory technology, dubbed Augmented Reality Interface for Submersibles (ARIS), for use with ROVs.

The project focuses on human-robot sensory transfer — what the National Science Foundation explains is the “seamless translation of perceptions and actions between the operator and the robot.”

Those perceptions are meant to improve worker productivity, training, and quality of life in remote undersea inspection and construction tasks. The project ticks off several of the International Maritime Organization’s sustainability design goals, such as using emerging technologies to support the emerging workforce.

The goal is to make it easier to operate ROVs, and to do so from a distance, according to Kevin McSweeney, manager of human factors, safety and emerging inspection technologies at ABS. 

Traditionally, he said, ROV operators have to be onboard a vessel to operate ROVs.

“But with this haptically-controlled suit, the ROV can be in the middle of the Gulf of Mexico, it could be in deepwater, and the suit can be in Gainesville, Florida, at the university,” he said. “When you experience rough or turbulent meteorological conditions, wave currents and such, you can very easily sense that these things are happening through the suit.”

Those sensations are far easier for an ROV operator to process than taking in information from dozens of gauges on various screens while trying to manipulate various controls. 

“It’s a virtual reality world where they can see everything that they need to. It’s really cool,” McSweeney said.

As designed, the ARIS full haptic feedback system involves haptic gloves, a haptic suit and a virtual reality (VR) headset like HTC VIVE, a video gaming technology. For small-scale ROV inspection, only the suit and headset are required, while the gloves are added to allow teleoperation of a work-class ROV’s robotic arms and receive force feedback.

The ROV pilot using the haptics stands rather than sits, and the suits are individually adjustable.

“All the information is provided to you through Unity, a physics engine used to construct the VR environment,” said Tianyu Zhou, a post-doctoral research associate at the University of Florida, who has been involved in the development of the haptic suit for ROV control.

Pengxiang Xia, a civil engineering doctoral candidate at the University of Florida who has been involved in the suit’s development, said the VR headset doesn’t limit the ROV operator’s spatial awareness.

“First, we’ll start off with just inspection, which is low-key. And then we look at moving into a place where we're actually using the actuators and turning knobs,” Xia said.

Zhou said the robotic arm has already been tested using haptic control and feedback.

Global connectivity

McSweeney said connectivity is one of the biggest concerns for remote operations of an ROV.

“Communication with the main asset or the onshore asset is one of the bigger challenges that we have,” he said.

Starlink satellite service and 5G cellular both play a role in connectivity.

Recently, McSweeney said, ABS carried out a remote operations test where control was done from Houston — digitally rather than through the haptics currently in testing. The ROV was in Norway. The results were transmitted to Brazil.

“That was all via 5G, and it worked very well from an ABS remote survey perspective,” he said. “Connectivity is getting better, but it is still a big challenge.”

One of the constants in remote operations is the lag time between a command being issued, and it being carried out. 

Xia said lag time with haptics should be similar to what traditional ROV operators are familiar with. In fact, haptic feedback is easier to transmit than video content, he said.

As McSweeney put it, “There will be some lag, but there’s also lag with traditional operations as well.”

The project has a number of industry partners, and ARIS is in trials aimed at qualifying the suit to Technology Level Readiness 3, a measurement of the advancement of the technology that is relatively low. 

“We need to evaluate the technology, to make sure that it can say it can do what it's being proposed to do,” McSweeney said.

A prototype is expected in December 2023, and once that’s available, he said, it will be easier to engage the industry in discussions about the technology.

“We're trying to understand the different operational requirements because when they have ROV operators out on an asset, you're talking about 12 hours on, 12 hours off. Very long shifts,” he said. 

The haptic suits make it “much easier” to control ROVs, he added.

McSweeney said the expectation is ROV pilots would use the new haptic suit for short spells, such as an hour or two at a time, until there is more understanding about the impact of haptics on human fatigue and task performance. 

The ARIS technology is appealing, he said, because it can help remove ROV operators from having to be offshore to do their jobs.

“If we're removing them from an offshore asset, we're reducing the number of people that have to be on board. So the operational costs are going to be cheaper for our clients and so on. It's potentially a very, very big win-win for everyone,” McSweeney said.