The industry is still adapting after two years of significantly depressed prices. On top of this, ‘the great crew change’ has meant a significant loss of experienced folks who understood processes and the business. These two factors have forced a technology transformation throughout the value chain to help reduce costs and get ahead of the competition.

Advanced analytics, enabled by open source technologies such as Apache Hadoop play a key part.

Hadoop is a software framework that helps companies process huge sets of data. It can find and land any and all data – structured, semi structured, and unstructured. This makes it an ideal place to converge the operational and IT worlds, enable powerful new visual and business intelligence tools, and advanced analytics such as machine and deep learning.

In the oil & gas industry, it is playing a major part in a company’s ability to ‘renovate’ older systems. IT organizations are looking at where they have duplicates, ‘shelfware’ that they can end of life, to offload licensing or reduce storage costs.

Almost all companies have some sort of old, outdated hardware gathering dust. This technology costs money to maintain, but still fills some need, whether it’s compliance or fear of downtime. Or it is just too costly to replace.

Many companies begin IT renovation by optimizing their data architecture. For example, they relocate cold data or freeing up processing from high cost legacy systems, or with enrichment programs to make existing data more valuable.

Paradoxically, oil and gas is also dealing with a huge growth in volume and complexity of data. Data is doubling every year. Much of this can be attributed to the higher velocity time series data coming from newly deployed sensors on operated assets that they wish to collect, process and analyze in order to maintain production and reduce unplanned downtime.

One innovation area that’s popular is a ‘single view’ of an asset or a process. The ability to connect data from both legacy databases and new operational sources is now possible within a single Open Source connected architecture. Historically those two worlds have been highly segregated.

In one case, an asset team from an upstream operator struggled for a year working with their IT department to consolidate all of the pertinent data sets from both OT, IT, and external sources to create a single view of their basin in legacy applications. By going with Hadoop, the company was able to identify potential failures in their value-chain four to five times faster.

Additionally, new regulations from BSEE requires offshore drillers to monitor safety critical equipment in real-time and archive the data onshore. Offshore drill ships are sparsely manned and operate in remote places with minimal bandwidth. Conventional remote monitoring technologies do not perform well under these conditions and large volumes of data ends up stranded offshore.

Using open source technology, in 90 days a company was able to aggregate, prioritize, compress and encrypt control system data from the ship’s equipment and transmit it over a 64 KB/sec satellite link to shore, providing real-time operating status to remote personnel.

A lot of energy companies aren’t sure where to start, the following is a set of recommendations.

  • Start small. Don’t spin up a “data lake” just for sake of it. Find a business challenge and work out a use case that will address it. It is absolutely vital to succeed or fail fast, and to find quick wins to demonstrate the transformation that’s possible with open source and analytics. For example, one company we worked with had challenges with visibility of inventory. Based on analyzing just 10% of data from their inventory systems, in just a few weeks it was able to find close to $4m in parts on the shelf that could be used to fix existing open orders. The short time-to-value shown from their initial use case helped launch its companywide analytics program.
  • Have a vision. For many in this industry, the goal is to get to prescriptive analytics. While predicting an event is ‘about to happen’ is important, the ultimate achievement is to then take that information and prescribe the necessary action to prevent it. To do so, organizations need to analyze data in real time data using the models built from analyzing past events—off of the well, the rig, the pipeline, refinery—and shifting from reactive to proactive.
  • Embrace open source. Open Source isn’t some newly hatched idea that people are frightened of any more. Plus, people have found that Open Source delivers functionality legacy technologies cannot, based on the advantages of an entire community of developers innovating and driving new functionality with transparency. This is a breath of fresh air for oil and gas companies that are used to using closed, proprietary technologies with little visibility into the development of new features and functionality.

Open source Apache Hadoop, combined with advanced analytics, are helping oil and gas companies lower costs, increase efficiencies, and knock down data silos to enable collaboration and data-driven intelligence via advanced analytics; delivering value across all lines of business. Open Source makes it easier.

Kenneth Smith is the General Manager for Energy at Hortonworks, where he is responsible for driving adoption across the energy industry. Hortonworks creates, distributes and supports enterprise-ready open and connected data platforms and modern data applications that deliver actionable intelligence.

Comments