Raising the Bar for ROV Operator Performance by Lowering the Bar for Entry

The robot revolution brought a mountain of good to the oil and gas industry, but also an increasing level of technical complexity. New ROVs and other types of emerging subsea technologies require their operators to be properly trained to carry out inspection, maintenance and repair tasks effectively. Current ROV operators may not be experts in working with new technologies, but training them would take too long and take away from their ROV operating time.

Greensea Technologies and Blue Ring Imaging recognized this problem and wanted to rectify it.

Originally partnered up while working with the U.S. Navy, both Greensea and Blue Ring brought their own tech to the table: Greensea’s OPENSEA platform provides navigation, autonomy and command and control, while Blue Ring’s augmented reality (AR) and virtual reality (VR) visualization application, “OctoView,” provides enhanced situational awareness for unmanned systems operators and teams.

“The future of our industry is all about autonomy. It's all about doing more with robots, with subject matter experts, not robot operators. And that is a profound concept, because our industry has grown up with professional robot pilots, professional ROV pilots,” Ben Kinnaman, CEO of Greensea Technologies, told Hart Energy. 

But, he said, the industry can't scale to all of its requirements due to the need for highly trained and skilled robot pilots.

“If we remove that blockage and we lower the bar of entry for operators, and we allow subject matter experts and engineers, first responders, enlisted operators and our armed services to use robots as tools, we can address all of the emerging needs of our industry,” Kinnaman said. “What OPENSEA does is knock down all of the barriers of adoption.”

OPENSEA and OctoView

The OPEN Software and Equipment Architecture, or OPENSEA, is Greensea’s comprehensive open architecture software platform for developing robotic systems intended to make it easier for people to use ocean robots. With OPENSEA, Greensea can provide customizable complex robotic systems to address real-world applications.

The OPENSEA platform includes a flexible navigation and localization engine with multiple navigating modes for different conditions and tasks. OPENSEA’s vehicle control architecture has adaptable autopilot controls to provide the ability to overcome environmental challenges, while a state-of-the-art autonomy platform combines task automation and perception technologies to assist, augment or even replace real-time human operation.

“Our goal is to reduce cognitive load and increase the user experience, enhancing [the] performance of these pilots when manual control is needed and when the operator has to take over.” – Casey Sapp, Blue Ring Imaging

Integrated inertial navigation, vehicle control, automated tasks, payload and target-relative positioning applications combine to provide an easy-to-use system so robot operators can focus on the tasks they must carry out rather than focusing on piloting the ROVs, says Kinnaman. 

When paired with Blue Ring, the OPENSEA platform creates an “easy path for a revolutionary technology like Blue Ring to reach the end user,” Kinnaman said.

Blue Ring Imaging’s goal is to make robot operations less complicated, CEO Casey Sapp told Hart Energy.

“Our goal is to reduce cognitive load and increase the user experience, enhancing performance of these pilots when manual control is needed and when the operator has to take over. How do we reduce that total time of operation and reduce complexity for both new and experienced pilots? We are hyperfocused on the operator experience complementing the great work Greensea does with vehicle control, mission planning and autonomy,” Sapp said.

Blue Ring’s OctoView is a vehicle, sensor and control system-agnostic app connected to OPENSEA, while its OctoCam is an extensible multi-lens camera that integrates with unmanned systems for enhanced situational awareness. OctoView, which is run through an AR or VR headset, pulls information from OPENSEA and remaps it in a 3D space, providing symbology and 3D imagery to perform visually-aided navigation.

The headset not only uses grids and markers to define how far underwater it is, it allows users to see location in the real world, said Sapp. Utilizing spatial information and special 3D markers, pilots can more quickly understand their environment and context to perform their job.

Avoiding context switching

Blue Ring’s aim for OctoView is to immediately help the operator perform inspections, surveys and robotic tasks in the simplest most time-efficient manner while increasing vehicle safety. Sapp’s ultimate goal is for the application to become a universal interface for all unmanned vehicle control.

“People are just like computers with how we spend time. We work well when we focus on one task. But, when we switch tasks, we lose time. The amount of time we spend switching tasks, that's the hard thing,” Kinnaman said. “For an ROV operator to develop and understand his situational awareness, he has to extract the information he needs from a dozen different sources by switching tasks between sources of information.”

As an example, Kinnaman said the operator might need to repeatedly switch between consulting multiple cameras, a computer screen and a mission plan to obtain information critical to situational awareness.

“Every time that operator moves his eyes from one screen to the next, he context switches. It takes time to do that. He loses relevance, he loses information and he has to reset his mind into a new task,” Kinnaman said. “What [Blue Ring’s] technology gives us is the ability for an operator to never context switch. He never has to leave the task he's on to derive the situational awareness he needs to conduct a mission.”

“Greensea, we manufacture fertile soil. Companies like [Blue Ring], they make plants. Those plants can't grow without fertile soil, so we've partnered so that Blue Ring's plant can grow.”– Ben Kinnaman, Greensea

When matched with the OctoCam, the system becomes a powerful edge compute multi-camera solution able to color-correct and dehaze images with a glass-to-glass latency of 100 milliseconds. Glass-to-glass latency is the time it takes for something to go from the glass of a camera to the glass of a display, so the lower the latency, the quicker images are transmitted to the person behind the screen.

While the offshore energy sector uses a myriad of VR technologies for pre-mission planning, post-mission analysis and maintenance instruction, Blue Ring sets itself apart by focusing on controlling vehicles in real time.

“There's very few examples and almost zero precedent of operational AR/VR, which is real time low latency utilization of this technology, specifically for this controlling of unmanned vehicles,” Sapp said. Because of this, Blue Ring “doesn’t really have any competitors at this point, especially in the subsurface space,” as they are one of the first and few to implement AR and VR operationally.

The OPENSEA software package is on more than 3,000 underwater vehicles and used by 35 manufacturers in the ocean robotics industry. Greensea has developed a generic interface between itself and Blue Ring, creating a direct path to use OctoView on any vehicle using OPENSEA.

Providing fertile soil

Blue Ring has been working closely with Greensea through a Navy Phase II SBIR to integrate the OctoView app on VideoRay Defenders for the explosive ordinance disposal ROV fleet, while the OctoCam should reach mass market in late 2023 or early 2024. Both companies are looking forward to a future where the barrier of adoption for ocean robots is lower and products such as headsets are used to increase situational awareness during missions.

“I think one way of thinking of it is to use a metaphor: Greensea, we manufacture fertile soil. Companies like [Blue Ring], they make plants. Those plants can't grow without fertile soil, so we've partnered so that Blue Ring's plant can grow,” Kinnaman said. “I can enable a better operating experience for robots by providing the fertile soil companies like Blue Ring can use to grow their own seed and reach those end users.”